Backpropagation

Forward pass, chain rule, computational graph, vanishing gradients

You have a network with a hundred million weights, and one training example. Running it forward to get a prediction is fast. But now you need to nudge *every one* of those hundred million weights in the right direction to make the prediction better.

Pause here: how would you compute the gradient for just one of those hundred million weights, without re-running the whole network? The naive idea is exactly what it sounds like — tweak that weight a hair, re-run the whole forward pass, see if the loss dropped, put it back, move on. That costs a full forward pass *per weight*. A hundred million forward passes for a single training step. At that price, deep networks would be untrainable, and for decades people were not sure they could be trained at all.

Backpropagation is the trick that makes it cheap. The key realisation: the forward pass already computed everything you need to work out *all* the gradients — you just have to reuse those cached values, walking backward through the network. One forward pass, one backward pass, and you have the gradient for every weight at once, no matter how many there are.

Watch it work (the network below)

Feed it x₁ = 1.0, x₂ = 0.5, with a hidden layer of two ReLU units [ReLU(z) = max(0, z) — pass positive inputs through unchanged, zero out negative ones] (W₁ = [[0.5, −0.3], [0.2, 0.8]], b₁ = [0.1, −0.1]) and a sigmoid output (W₂ = [0.7, −0.5], b₂ = 0), against a target of 1.0. Forward: the hidden layer computes z₁ = [0.45, 0.5] — both positive, so ReLU passes both through unchanged to a₁ = [0.45, 0.5] — the output layer computes z₂ = 0.065, and sigmoid squashes that to a prediction of about 0.516. The loss is mean squared error, L = (prediction − target)² — against a target of 1.0, that's L = (0.516 − 1.0)² ≈ 0.234. That squaring is exactly where the ×2 in the next step comes from: d/dprediction of (prediction − target)² is 2(prediction − target).

Now go backward. Picture the gradient as a message passed down through the floors of a building: the loss sits on the top floor and hands instructions to W₂ first, one floor down, then further down to W₁, one more floor below. Pause and predict before reading on: which of the two ends up with the *smaller* gradient — W₂, right next to the output, or W₁, one extra handoff away — and why?

Walk the chain rule backward, naming each handoff as you go. The output floor's error signal is δ₂ = (prediction − target) · sigmoid′(z₂) · 2 ≈ −0.242 — that's the message sigmoid hands down. From there, dL/dW₂ = δ₂ · a₁ = [−0.109, −0.121], averaging about −0.115. One more hop down: the hidden floor's error signal is δ₁ = δ₂ · W₂ · ReLU′(z₁) = [−0.169, 0.121]. Since both of z₁'s entries are positive, ReLU′ is 1 for each one, so this hop barely shrinks anything — δ₁'s two entries are the same order of magnitude as δ₂, not a fraction of it. That gives dL/dW₁ = δ₁ · x = [−0.169, −0.085, 0.121, 0.060]. Here's the part that's genuinely surprising: those four entries aren't all the same sign — two are positive, because W₂'s second weight is negative and flips the sign of that hidden unit's contribution. Average them with their signs intact and you get about −0.018, roughly six times smaller than W₂'s −0.115. That shrinkage has nothing to do with the ReLU hop itself (which passed the error signal through essentially unchanged) — it comes from two of the four entries partly cancelling each other out. The one hop that *did* shrink the signal was the output floor's sigmoid — its slope there was about 0.25, the only toll the whole backward trip paid; ReLU's slope of 1 charged nothing.

Why deep networks stalled: vanishing gradients

Now imagine that hidden floor had been sigmoid instead of ReLU. It would have charged the same toll the output floor just did — about 0.25 — and tolls *compound*. A sigmoid's slope is *at most* 0.25, and usually less. Multiply that in at every handoff, and after 10 sigmoid floors the message reaching the first layer is scaled by roughly 0.25¹⁰ — about one in a million; after 20, essentially zero. The instructions sent from the top floor arrive at the bottom as silence. The early layers get no signal and never learn, so your "20-layer network" quietly behaves like a 2-layer one. That's exactly what the ReLU floor above avoided — its slope of 1 charges no toll at all, which is why swapping sigmoid hidden layers for ReLU is the fix. (ReLU's own catch is the *dead neuron* — one whose input is always negative outputs zero forever, dropping out of the relay; Leaky ReLU and GELU keep a trickle of gradient flowing to prevent that.)

The mirror-image failure is exploding gradients, common in recurrent networks that multiply the same weights over and over: if that factor is even slightly above 1, the message doesn't shrink to zero — it blows up to NaN instead. The fix is gradient clipping — if the gradient's overall size exceeds a cap, scale it back down while keeping its direction. And residual connections fix vanishing structurally: add a shortcut (output = layer + input) so the message gets a direct path back that skips the shrinking handoffs entirely, which is why ResNets train hundreds of layers deep.

One practical cost: memory

Because the backward pass reuses the forward pass's intermediate values, they all have to be *stored* until the backward pass runs — which for a big model is a lot of memory. Gradient checkpointing is the standard trade: keep only a few of those intermediates and recompute the rest on the fly during the backward pass, spending about 30% extra compute for O(√depth) memory. It is what lets large models train on limited GPUs. (And for the curious: backprop is often called "just the chain rule," but the magic is applying it in *reverse* order — that reverse direction is exactly what makes the whole thing cost one forward pass instead of one-per-parameter.)

One more knob: batch size

Everything above walked through a single training example. In practice you compute the loss over a *batch* of examples at once, and backprop still applies unchanged per example — the batch gradient is the average of the per-example gradients, not their sum. That means a batch of 32 and a batch of 1 produce gradients of roughly the same *magnitude*: averaging doesn't shrink or grow the signal, it lowers its variance — 32 noisy individual estimates settle toward the same expected value one estimate already targets, just less reliably. What batch size changes is how confidently you can take a bigger step: the standard practice, the linear scaling rule, is to scale the learning rate in proportion to the batch size — going from batch 32 to batch 256 (8× larger) calls for roughly 8× the learning rate — to keep training dynamics comparable across batch sizes.

ReLU stops the message from shrinking on the way down — but it trades one failure for another, the dead neuron. Working out exactly when a neuron dies, and how Leaky ReLU and GELU keep the relay alive, is where the next module, Activation Functions, picks up.

Key points

• When your network has more than one layer and you need to update weights end-to-end. That is always. Backprop is the only practical algorithm for computing exact gradients in deep networks. You do not implement it yourself — every modern framework (PyTorch, JAX, TensorFlow) runs it automatically. What you do need to understand: the forward pass must cache intermediate activations, gradient checkpointing trades 30% extra compute for O(√depth) memory (mandatory for large models on limited GPU), and the gradient accumulates by summation when multiple paths lead to the same node.
• The production trap: ignoring gradient norms. Backprop produces the correct gradient mathematically, but "correct" can be a gradient of 10⁻¹² — numerically zero. Early layers in deep sigmoid networks receive no learning signal, and training proceeds as if those layers do not exist. Always log gradient norms per layer during the first training run. A ratio of 1000:1 between last-layer and first-layer norms means the depth is wasted. The fix is ReLU activations or residual connections, not more data or a different learning rate.
• The diagnostic: verify gradient flow before anything else. Register a backward hook on each layer and log the mean absolute gradient at each step. For a 10-layer network with healthy gradient flow, the norms should decay by at most ~10× from output to input — not 10⁶×. If you see exponential decay, the activation function is saturating. If you see exponential growth, gradient clipping (max_norm=1.0) is missing. Both symptoms are visible before the first epoch completes.

Recap

• Naive gradient = one forward pass per weight: tweak a weight, re-run the whole forward pass, see if loss dropped, repeat — with 100M weights that's 100M forward passes for a *single* training step. At that price deep nets are untrainable, which is why for decades people doubted they could be trained at all.
• Backprop = one forward + one backward pass total: the forward pass already computed everything the gradients need, so you cache those intermediate values and walk backward once, recovering the gradient for *every* weight at once — no matter how many there are.
• The chain rule in reverse is the magic: backprop is "just the chain rule," but applying it in *reverse* order (output back to input) is what collapses the cost to one pass instead of one-per-parameter. Gradients also *sum* when multiple paths reach the same node.
• Vanishing gradients: a sigmoid's slope is at most 0.25, multiplied in at every layer → ~0.25¹⁰ ≈ one-in-a-million after 10 layers, essentially zero after 20. Early layers get no signal and never learn, so a "20-layer net" behaves like a 2-layer one. Fix: ReLU (slope 1 for active neurons) and residual connections (a shortcut that skips the shrinking multiplications).
• Exploding gradients: the mirror failure — common in RNNs that multiply the same weights repeatedly. If the factor is even slightly above 1, the gradient blows up to NaN. Fix: gradient clipping — if the gradient's overall size exceeds a cap, scale it back down while keeping its direction.
• Memory cost: because the backward pass reuses the forward pass's intermediate activations, they must all be *stored* until it runs — a lot of memory for a big model. Gradient checkpointing keeps only a few and recomputes the rest on the fly: ~30% extra compute for O(√depth) memory, which is what lets large models train on limited GPUs.
• Diagnostic — log per-layer gradient norms on the first run: a healthy 10-layer network's norms decay by at most ~10× from output to input. A 10⁶× (1000:1+) ratio means the activation is saturating and the depth is wasted; exponential *growth* means clipping (max_norm=1.0) is missing. Both are visible before the first epoch finishes.

Check your understanding