Gradient Clipping and Regularization

Bounding gradient explosions, weight decay vs L2, dropout, and connecting optimizer choices to generalization.

A big neural network has enough raw capacity to simply *memorise* its training data — to store the answer key rather than learn the pattern. Left unchecked, it will do exactly that: acing training, flopping on anything new. This lesson is about the toolkit that stops it, plus one safety valve for a different problem entirely.

The safety valve: gradient clipping

First, the odd one out. Gradient clipping is not really about memorisation — it is a seatbelt. As we saw with exploding gradients, sequence models can occasionally produce a gradient thousands of times bigger than usual, and a single such step can wreck the whole model. Clipping simply says: if the gradient's overall size exceeds a cap, shrink it back to the cap while keeping its *direction*. It prevents catastrophe; it does nothing for generalisation. (Clip by the whole-vector size, not each component separately, or you bend the direction — norm-clip at 1.0 is standard for language models.)

The main tool: weight decay

The workhorse regulariser is weight decay: at every step, gently shrink every weight toward zero by a small fixed fraction. This is the same "keep the weights small so the model stays simple" idea from the regularisation lesson — small weights mean a smoother, less memorising model.

There is one subtlety that trips up nearly everyone. For plain SGD, weight decay is *exactly* the same as adding an L2 penalty to the loss. For the Adam optimizer, it is *not*. Adam rescales each parameter's update by how big that parameter's gradients have been; fold the L2 penalty into the loss and it gets rescaled too, so parameters end up regularised *unevenly* — and backwards from what you want. The fix is AdamW, which applies the weight decay *directly* to the weights, off to the side of Adam's rescaling, so every weight shrinks by the same fraction as intended. This one change measurably improves generalisation, which is why AdamW — not Adam — trains modern language models. If you take one practical thing from this lesson: with Adam, use AdamW, never L2-in-the-loss.

Two more, for two more failure modes

Dropout attacks over-reliance. During training it randomly switches off a fraction of neurons on each step, so the network can never lean on any single neuron or pathway — it is forced to build redundant, backup representations. At test time every neuron is back on. It is heavy for plain fully-connected layers (drop about half) and lighter in transformers (drop 10–30%), where attention already spreads things out.

Label smoothing attacks over-confidence. Normally the training targets are hard 0s and 1s, which push the model to become *infinitely* confident to drive the loss to zero — and wildly overconfident models are poorly calibrated. Label smoothing softens the targets slightly (say 0.9 instead of 1.0), making perfect confidence impossible and keeping the model's probabilities honest.

They are a system, not a checklist

The catch is that these tools interact — with each other and with the optimizer. Small-batch SGD already injects noise that regularises for free, so a small model trained that way may need little else; large-batch Adam has almost no built-in noise and leans hard on explicit weight decay and dropout to compensate. Pile on every regulariser at full strength and you can *over*-regularise a small model into underfitting. So there is a budget: tune one knob at a time, start with weight decay (the biggest lever), and match the amount to your model size and dataset size rather than reaching for all of them at once.

Key points

• Gradient clipping is a safety valve, not a regulariser. Sequence models can occasionally throw a gradient thousands of times too big, and one such step can destroy the model. Clipping caps the gradient's overall size while keeping its direction, so no single step is catastrophic. Clip by the whole-vector norm, not per-component (per-component clipping bends the direction) — norm-clip at 1.0 is the standard for language models. It prevents disasters; it does nothing for generalisation.
• Weight decay is the main regulariser — but with Adam you must use AdamW, not L2 in the loss. Weight decay shrinks every weight toward zero by a small fixed fraction each step, keeping the model simple. For plain SGD this is identical to an L2 penalty. For Adam it is not: an L2 term folded into the loss gets rescaled by Adam's per-parameter scaling and comes out uneven and backwards. AdamW applies the decay directly to the weights, off to the side of that rescaling, so it shrinks every weight uniformly as intended — which is why AdamW trains modern language models. Rule of thumb: with Adam, always AdamW.
• Two more tools for two more failure modes: dropout for over-reliance, label smoothing for over-confidence. Dropout randomly switches off a fraction of neurons during training, so the network cannot depend on any single one and must build redundant backups (drop ~50% in dense layers, 10–30% in transformers; all neurons return at test time). Label smoothing softens the hard 0/1 targets slightly, making perfect confidence impossible — which stops the model becoming wildly overconfident and keeps its probabilities well-calibrated.
• Regularisers are a coupled system with a budget — do not just stack them all. Small-batch SGD already regularises for free through its gradient noise, so a small model may need little more; large-batch Adam has almost none and leans on explicit weight decay and dropout. Stack every regulariser at full strength and you can over-regularise a small model into underfitting. Tune one knob at a time, start with weight decay (the largest lever), and match the total to your model size versus dataset size.

Recap

• Gradient clipping is a safety valve, not a regularizer: it caps the gradient's overall size while keeping its direction, so no single catastrophic step wrecks a sequence model (RNNs especially) — norm-clip at 1.0. Calling it regularization in an interview is a red flag.
• Weight decay is the primary regularizer: shrink every weight toward zero by a fixed fraction each step, keeping the model simple and its solution flat — the biggest single lever you have.
• With Adam, always use AdamW, never L2-in-the-loss: Adam's per-parameter √v̂ rescaling makes an L2 penalty uneven and backwards (it decays the wrong weights hardest); AdamW applies the decay directly and uniformly to the weights.
• Dropout attacks over-reliance / co-adaptation: randomly switch off neurons each step (~50% in dense layers, 10–30% in Transformers) so the network can't lean on any one path and builds redundant representations.
• Label smoothing attacks over-confidence: soften the targets (0.9 instead of 1.0) so the model can never drive its logits to infinity, which improves calibration and stops it from being overconfident on ambiguous inputs.
• Regularizers are a coupled system with a budget, not independent knobs: small-batch SGD already regularizes for free via gradient noise, while large-batch Adam has almost none and leans on explicit decay/dropout — stack everything at full strength and you can over-regularize a small model into underfitting.
• Tune one knob at a time, weight decay first (the largest lever), and match the total regularization to your model size versus dataset size rather than blindly maxing every technique.

Check your understanding