Batch Normalisation & Regularisation

BatchNorm vs LayerNorm — why normalisation stabilises training, and why it comes with free regularisation

Choosing the right activation fixed how gradients flow *through* each layer — but it says nothing about what a layer's *inputs* look like from one training step to the next. Take a 10-layer network mid-training. Layer 5 tweaks its weights — fine. But layer 6 had learned to expect a certain *distribution* of numbers coming from layer 5, and that distribution just moved. So layer 6 scrambles to adjust, which shifts what layer 7 sees, and so on up the stack. Every layer is chasing a moving target created by the layers below it. To keep this from blowing up, you are forced to use a tiny learning rate so no single update destabilises everything above it — and training crawls. This wobble was one of the big reasons deep networks were so fragile to train before 2015.

Batch normalisation fixed it with a simple idea: at each layer, before passing the numbers on, *re-centre and re-scale them* so they have a consistent, tidy distribution (mean 0, spread 1 — using the batch's own mean μ_batch and spread σ_batch) across the batch. Now layer 6 always sees inputs in a familiar range no matter what layer 5 did, and the moving-target problem largely goes away. (It also keeps a pair of learned dials, γ and β, that let the network re-stretch the numbers if the task actually needs it, so nothing is lost.) The payoff is big: the original paper (Ioffe & Szegedy, 2015) reports raising the learning rate by roughly 5× in their main experiment — and up to 30× in a more aggressive variant — above what worked without it, the network stops caring so much about initialisation, and training converges much faster. (Later work — Santurkar et al., 2018 — argued this speed-and-stability payoff comes less from taming the moving-target problem and more from smoothing the loss landscape, making the optimisation surface easier to descend regardless of which story you tell about the mechanism.)

A happy side effect: free regularisation

Here is a subtlety that turns out to matter. The mean and spread used to normalise are computed from the *current mini-batch* — so the exact same example gets normalised a little differently depending on which other examples happen to share its batch. That tiny, ever-changing jitter acts like a mild regulariser: the network cannot lean too hard on any one example's exact representation, because that representation keeps shifting.

Batch norm vs layer norm — not interchangeable

There is a second normaliser, layer norm, and picking the wrong one is a genuine error, not a tuning choice. Batch norm normalises each feature *across the batch* — which only makes sense if the examples in a batch are comparable. In a Transformer chewing through tokens from different positions in different sentences, "the average of this feature across the batch" is semantic nonsense. Layer norm instead normalises *across the features of a single example*, so it is well-defined for one token at a time, at any position, with any batch size. That is why every Transformer uses layer norm, and CNNs on images use batch norm.

(One practical gotcha with batch norm: at inference you have no batch, so it switches to running averages collected during training. Forget to flip the model into eval mode and a single-example prediction gets normalised against a batch of one — which quietly produces garbage, with no error.)

Key points

• Batch norm for CNNs on images (batch size ≥ ~8 comparable examples); layer norm for Transformers, RNNs, and any variable-length or small-batch task. The choice comes from what the statistics *mean*, not from tuning. Batch norm needs a batch of at least ~8 comparable examples or its per-batch estimates are too noisy to help. And always switch the model to eval mode at inference, so it uses the running averages instead of a (possibly size-1) batch — forgetting this is the single most common silent failure in deployed vision models.
• The trap: batch norm's train-versus-inference mismatch. During training each example is normalised using its mini-batch's mean and spread; at inference the model uses running averages collected during training. If the input distribution shifts — new data source, different camera, different preprocessing — those stored averages are stale and the normalisation is wrong, and accuracy degrades with no error, no NaN, no warning. Fix it by running a few forward passes over data from the new distribution (in train mode) to refresh the running statistics before switching back to eval.
• The diagnostic: compare training loss at a large batch versus a small one. If the same model trains noticeably worse and noisier at batch size 4 than at 32, batch norm is the culprit — with only four samples the per-batch mean and variance are poor estimates and destabilise the normalisation. Swap in group norm or layer norm and re-run; if the gap closes, that confirms it.

Recap

• The problem — a moving target: when a layer updates its weights, the distribution of numbers it emits shifts, so the layer above (which had learned to expect the old distribution) must scramble to re-adjust — and so on up the stack. Every layer chases a moving target, forcing a tiny learning rate so no update destabilises everything above it. Training crawls.
• Batch norm: before passing numbers on, re-centre and re-scale each layer's outputs to mean 0, spread 1 *across the batch*, so the next layer always sees a familiar range. A learned pair of dials (γ scale, β shift) lets the network re-stretch if the task actually needs it, so nothing is lost.
• Payoff: the original paper (Ioffe & Szegedy, 2015) reports raising the learning rate by roughly 5× in their main experiment, up to 30× in a more aggressive variant, the network stops caring so much about initialisation, and training converges much faster — this is what made pre-2015 deep nets far less fragile. (Later work — Santurkar et al., 2018 — argued this speed/stability payoff comes less from taming the moving target and more from smoothing the loss landscape.)
• Free regularisation: the mean/spread come from the *current* mini-batch, so the same example is normalised slightly differently depending on its batch-mates. That ever-changing jitter mildly regularises — the network can't lean on any one example's exact representation.
• Batch norm vs layer norm is a correctness choice, not a knob: BN normalises a feature *across the batch*, which only makes sense if the batch's examples are comparable — meaningless for tokens from different sentence positions. LN instead normalises *across a single example's features*, well-defined for one token at any position with any batch size. So CNNs on images use BN; Transformers, variable-length, and small-batch tasks use LN.
• Batch norm inference gotcha: at inference there's no batch, so BN switches to running averages collected during training. Forget to call `model.eval()` and a single-example prediction gets normalised against a batch of one — silently producing garbage with no error thrown.
• Diagnostic: if the same model trains much noisier and worse at batch 4 than at 32, BN's per-batch mean/variance estimates are too poor with few samples — swap in group norm or layer norm and if the gap closes, that confirms it.

Check your understanding