Gradient Flow

How gradients travel backward through deep networks — and why they vanish or explode.

For years, deep networks were a great idea that nobody could actually train. Stack more layers and, in theory, the network learns richer things — but in practice the deep ones just refused to learn. The culprit turned out to be something subtle: how the *learning signal* travels backward through the layers.

Here is the picture. To learn, the network sends an error signal from the output back toward the input, layer by layer, telling each layer how to adjust. But at every layer that signal gets *multiplied* by the layer's local sensitivity. Multiply a number by something a bit less than 1 over and over — 0.25 × 0.25 × 0.25 … — and it shrinks toward nothing astonishingly fast. Multiply by something a bit more than 1 over and over and it blows up just as fast. So a signal that starts healthy at the output arrives at the early layers as either a vanished whisper (near zero) or an exploded scream (enormous). Either way, learning breaks.

Vanishing gradients — the whisper that fades to nothing

The old sigmoid activation was the classic offender. Its local sensitivity is at most 0.25, and usually much less. Chain ten sigmoid layers together and the signal reaching the first layer is scaled by at most 0.25^10 — about a millionth of what left the output. The early layers get essentially *no* signal, so their weights barely move; the network learns only in its last few layers and wastes all its depth. This is exactly why deep sigmoid networks stalled.

The first fix was a better activation: ReLU, whose sensitivity is a clean 1 for any active neuron — it passes the signal through unchanged instead of shrinking it by a fraction. (ReLU has its own smaller catch: a neuron whose input stays negative outputs zero forever, a "dead" neuron. But it hugely reduced the vanishing problem.)

Exploding gradients — the whisper that becomes a scream

The opposite failure shows up especially in RNNs, which process a sequence by applying the *same* weights at every time step. Unroll one over 200 steps and the signal gets multiplied by that weight matrix 200 times. If the matrix is even slightly "bigger than 1" in effect, 1.05 to the 200th power is about 17,000 — the signal explodes, the weights leap to absurd values, and the loss turns to NaN. The quick, blunt fix is gradient clipping: if the signal's overall size exceeds some cap, shrink it back to the cap while keeping its *direction* the same. A seatbelt, not a cure.

The real cures were architectural

Clipping and ReLU helped, but the breakthroughs were changes to the architecture itself, all aimed at giving the signal a clean path home.

Residual connections (ResNets) add a shortcut around each block: the block's output is "what the block computes *plus* its own input." On the way back, that shortcut hands the signal a *direct* route to earlier layers that skips the multiplying entirely — so even a 100-layer network keeps a full-strength signal reaching layer 1. This one trick is why we can train networks hundreds of layers deep.

LSTMs did the same for sequences. Instead of *multiplying* the signal through every time step (which explodes or vanishes), an LSTM carries a running memory that mostly gets *added to*. When its "forget gate" says "hold on to this," the signal flows back across hundreds of steps almost untouched — a gradient highway through time — which is what finally let networks learn long-range patterns.

Layer normalization helps too, by rescaling each layer's activations so they stay in the healthy middle range of the activation function, where the local sensitivity is largest and the signal does not get crushed.

The through-line: ReLU, clipping, residual connections, LSTM gates, and layer norm are not fancy representational upgrades. They are all *plumbing* — machinery whose entire job is to make sure the learning signal survives the trip backward through a deep network.

Initialisation: the first line of defence

Before any of those fixes, the *starting* weights already decide whether the signal survives. If they're too large the activations explode; too small and they vanish — from step one. The principled fix is to scale the initial weights so the *variance* of the signal is preserved as it passes through each layer, in both the forward and backward directions. Xavier/Glorot initialisation does this for tanh/sigmoid (variance ≈ 1/fan_in, or 2/(fan_in+fan_out) to balance forward and backward). He/Kaiming initialisation is the ReLU version (variance ≈ 2/fan_in), the extra factor of 2 compensating for ReLU zeroing out half its inputs. Use He with ReLU, Xavier with tanh/sigmoid — the wrong one on a deep plain network reintroduces vanishing/exploding at initialisation.

BatchNorm versus LayerNorm

Both stabilise the signal by normalising activations, but along *different axes*. BatchNorm normalises each feature *across the batch* — it depends on batch statistics, which makes it powerful for CNNs but batch-size-sensitive and awkward for variable-length sequences. LayerNorm normalises *across the features within a single example*, so it's independent of batch size and of other examples — which is exactly why it's the norm of choice in transformers and RNN-like models where sequences vary and batch statistics are unreliable. The axis is the whole distinction: BatchNorm across the batch, LayerNorm across the features.

Fixing dead ReLUs

ReLU's "dead neuron" catch (input stuck negative → output zero forever → no gradient) has a family of fixes. Leaky ReLU gives a small negative slope so the neuron always passes *some* gradient. ELU and GELU (the transformer default) are smooth variants that keep a non-zero gradient for negative inputs. Beyond activations, better initialisation (He) and a lower learning rate reduce the chance a neuron gets pushed into the dead zone in the first place. If a large fraction of your ReLUs are dead (check activation histograms), that's the lever.

Transformers: residuals + norm, and where the norm goes

Everything above is *why* transformers are built the way they are. Each transformer block wraps attention and the feed-forward network in residual connections and layer normalisation — that combination is what keeps gradients flowing through dozens of blocks. And *where* you put the norm matters: post-norm (the original, norm after the residual add) is harder to train deep and needs careful warmup; pre-norm (norm inside the residual branch, before attention/FFN) gives a cleaner gradient path and trains stably at great depth, which is why modern large models use it. "Pre-norm vs post-norm" is a real interview question about gradient stability, not a detail.

Reading gradient flow: the diagnostics

You don't have to guess whether gradients are healthy — measure. Per-layer gradient norms: log the gradient magnitude at each layer; a healthy net keeps them within ~10× across layers, and a steep decay toward the input means vanishing. Activation histograms: watch for saturation (piling at the extremes) or lots of dead zeros. NaN/Inf checks: catch explosions the moment they appear. Weight-update ratios: the size of each step relative to the weight it updates should sit around 1e-3; far smaller means a layer is barely learning. These turn "training is off" into "layer 2 is vanishing."

Symptom map, and what clipping can't fix

Tell the two failures apart by their signatures. Vanishing: early-layer gradients near zero, those layers barely update, loss falls painfully slowly or stalls, and the network behaves shallow. Exploding: gradient norms blow up, loss oscillates wildly or goes NaN, weights leap to extremes. And be clear about clipping's limits: it's a seatbelt against catastrophic single steps, but it does *not* fix poor conditioning, a bad learning rate, bad initialisation, or slow long-term vanishing — if you're clipping on most steps, the real problem is upstream (LR too high, init wrong), and clipping is just masking it.

Key points

• The core mechanism: the backward signal is multiplied by each layer's sensitivity, so small factors vanish it and large ones explode it — exponentially with depth. Backpropagation sends the error from the output to the input one layer at a time, scaling it by each layer's local sensitivity as it goes. A chain of factors below 1 shrinks the signal toward zero (vanishing); a chain above 1 blows it up (exploding). The deeper the network, the more extreme it gets — which is why depth was so hard to train before the fixes below.
• Activations matter: sigmoid caused vanishing, ReLU mostly cured it. Sigmoid's sensitivity peaks at just 0.25, so ten layers can shrink the signal by a factor of a million and the early layers never learn. ReLU passes the signal through at full strength (sensitivity 1) for any active neuron, removing that per-layer shrink. ReLU's own catch is "dead" neurons — ones whose input stays negative and so output zero forever — but it was still the change that made deeper networks trainable.
• Exploding gradients hit RNNs hardest, and clipping is the seatbelt. An RNN applies the same weights at every step, so over a long sequence the signal is multiplied by that matrix again and again; if it is even slightly amplifying, the signal explodes and the weights blow up to NaN. Gradient clipping caps the overall size of the update while keeping its direction, so no single step is catastrophic. Clip by the whole-vector norm (rescale everything together), not per-component, or you distort the direction — norm clipping at 1.0 is the transformer default.
• The durable cures are architectural: give the signal a clean path home. Residual connections add a shortcut around each block, so the signal has a direct route back that skips the multiplying — which is why 100-plus-layer networks train. LSTMs carry a memory that is mostly *added* to rather than multiplied through, so when the forget gate says "keep this," the signal flows back across hundreds of time steps almost intact. Layer normalization keeps activations in the healthy middle range where sensitivity is highest. All of it is plumbing to keep the backward signal alive.
• Initialisation comes first, and the normalisation axis matters. Scale initial weights to preserve signal variance forward and backward: He/Kaiming (variance 2/fan_in) for ReLU, Xavier/Glorot (1/fan_in) for tanh/sigmoid — the wrong one reintroduces vanishing/exploding at step one. BatchNorm normalises each feature across the batch (batch-size-sensitive, great for CNNs); LayerNorm normalises across features within one example (batch-independent, the transformer/RNN choice). Fix dead ReLUs with Leaky ReLU / ELU / GELU, better init, or a lower learning rate.
• Diagnose gradient flow directly, and know clipping's limits. Transformers stack residual connections + layer norm, and pre-norm (norm inside the residual branch) trains deeper and more stably than the original post-norm. Measure health with per-layer gradient norms (a steep decay toward the input = vanishing), activation histograms (saturation/dead zeros), NaN checks, and weight-update ratios (~1e-3). Read the symptoms: vanishing = early layers stall, loss crawls; exploding = NaN, oscillating loss, huge norms. Clipping is a seatbelt against catastrophic steps — it does not fix bad conditioning, LR, or init, so clipping on most steps means the real problem is upstream.

Recap

• Core mechanism: the backward signal is multiplied by each layer's sensitivity as it propagates, so a chain of factors < 1 shrinks it toward zero (vanishing) and a chain > 1 blows it up (exploding) — and both compound *exponentially* with depth.
• Sigmoid caused the vanishing era: its sensitivity is ≤ 0.25, so $0.25^{10}$ ≈ a millionth reaches the first layer of a 10-layer net. ReLU mostly cured it with sensitivity 1 for active neurons, passing the signal through undiminished.
• Exploding gradients hit RNNs hardest because they multiply the *same* weight matrix T times through the sequence — even a factor slightly above 1 detonates to NaN. Gradient clipping caps the update's overall size while keeping its direction: a seatbelt against one catastrophic step.
• The durable cures are architectural, not tuning: residual connections give the gradient a direct identity route home, LSTM gates *add* to the cell state rather than repeatedly multiply, and layer norm keeps activations in the responsive range where sensitivities stay near 1.
• Initialization comes first — it's gradient flow at step zero: He (variance 2/fan_in) for ReLU, Xavier (1/fan_in) for tanh/sigmoid. The wrong recipe reintroduces vanishing or exploding from the very first forward pass, before the optimizer even runs.
• Norm axis is a correctness choice: BatchNorm normalizes a feature *across the batch* (batch-sensitive, used in CNNs); LayerNorm normalizes *across features* within one example (batch-independent, used in Transformers).
• Diagnose gradient flow directly: log per-layer gradient norms (a steep decay toward the input = vanishing), activation histograms (saturation / dead zeros), NaN checks, and weight-update ratios (~1e-3). If clipping fires on most steps, the real problem is upstream — bad conditioning, LR, or init — clipping won't fix those.

Check your understanding