Backpropagation: What the Chain Rule Is Actually Doing

Most explanations of backprop describe the algorithm. This one starts with why it has to work this way. A neural network is a composed function. Training it means finding how each weight contributes to the final loss. The chain rule is the only tool that can answer that question efficiently. Once you see it as function composition and gradient routing, backprop stops being magic.

A neural network is a composed function. You feed an input forward through a series of transformations — linear projections, activations, more linear projections — and at the end you compute a scalar loss. Training is the process of adjusting every weight in that chain so the loss gets smaller. Backpropagation is how you compute the gradient of that loss with respect to every weight. The chain rule is the only tool that makes it tractable.

What the forward pass is actually doing

Consider the simplest possible network: two layers, one activation. The computation is: z1 = W1 * x + b1, then a1 = relu(z1), then z2 = W2 * a1 + b2, then loss = cross_entropy(z2, y). Each step is a function applied to the output of the previous step. The entire network is a composed function: loss = f4(f3(f2(f1(x)))). During the forward pass you compute each intermediate value and store it. This stored state is not wasted memory — it is required for the backward pass.

The chain rule: one variable at a time

To find how W1 affects loss, you need d(loss)/d(W1). The chain rule says: if loss depends on z2, z2 depends on a1, and a1 depends on W1, then d(loss)/d(W1) = d(loss)/d(z2) * d(z2)/d(a1) * d(a1)/d(z1) * d(z1)/d(W1). That product of four terms is backpropagation applied to this network. Each term is a local gradient: how much does this node's output change if its input changes? Local gradients are cheap to compute and only require information available at that node.

The computational graph and gradient routing

In the general case, a network is a directed acyclic graph of operations. Each node computes a function of its inputs and produces an output. During the backward pass, gradients flow backward through the same graph. The rule at each node: multiply the gradient arriving from downstream by the local gradient, and route the result upstream to each input. If a node has multiple outputs feeding into different downstream nodes, the gradients from all downstream paths are summed before being routed further upstream. This sum is the total contribution of that node to the loss across all paths through the graph.

This is why stored activations from the forward pass are necessary. To compute d(a1)/d(z1) at the relu node, you need to know whether z1 was positive (gradient = 1) or negative (gradient = 0). You stored z1 in the forward pass to answer exactly this question during the backward pass.

Why ReLU fixed vanishing gradients

With sigmoid activations, the local gradient is sigmoid(x) * (1 - sigmoid(x)), which has a maximum of 0.25. In a deep network with 10 layers, the gradient arriving at the first layer is the product of 10 such terms — at most 0.25^10, roughly 10^-6. The gradient signal vanishes before it reaches the early weights, which therefore learn nothing.

ReLU's local gradient is 1 for positive inputs and 0 for negative inputs. The product of 10 terms of 1 is still 1. The gradient passes through unchanged wherever neurons are active. Early layers now receive a meaningful gradient and can learn. The dead neuron problem (zero gradient for all inputs) is real but manageable — and a worthwhile trade for gradient flow.

What it means for a weight to have a large gradient

d(loss)/d(W) = 0.8 means: if you increase W by a small amount ε, the loss increases by 0.8ε. The gradient tells you both direction (which way to move W) and magnitude (how sensitive the loss is to that weight). Gradient descent subtracts a fraction of this gradient from each weight, moving it in the direction that decreases loss. A weight with near-zero gradient is either not contributing to the loss (a candidate for pruning) or stuck in a flat region (needs a better initialisation or learning rate schedule).

Depth and function composition

Why does depth help? Composing functions lets the network learn features hierarchically. The first layer learns low-level structure; subsequent layers combine those into more abstract representations. Backprop propagates credit assignment through this entire hierarchy. Without it, only the last layer could be trained directly — everything else would have to be hand-engineered.

The Jacobian of a composed function is the product of the Jacobians at each layer. For this product to be informative — not vanishing and not exploding — the Jacobians need magnitudes close to 1. This is the motivation for batch normalisation, residual connections, and careful initialisation. All three are engineering solutions to the same mathematical problem: keeping gradient products well-conditioned through many layers.

Try on Colab: implement a 2-layer network in raw NumPy. Write the forward pass explicitly, then hand-code the backward pass using the chain rule derivations above. Train on a 2-class synthetic dataset. Compare your weight updates step-for-step against PyTorch autograd on the same network. They should be numerically identical to floating-point precision.