Weight Initialization

Symmetry breaking, Xavier and He initialization, and the connection to gradient flow.

Before a network takes a single training step, the *starting* values of its weights already decide whether it can learn at all. Two things can go wrong at the very beginning, and both are worth understanding.

The first is a surprise: do not set all the weights to zero. It sounds harmless, but it is fatal. If every neuron in a layer starts with identical weights, they all compute the same output, all receive the same gradient, and all take the same update — so they stay identical forever. A 512-neuron layer initialised to zero behaves exactly like a *single* neuron; the other 511 are wasted. The fix is random initialisation: the randomness is what makes neurons different from one another so they can learn different features. This is called breaking symmetry.

The second problem: getting the scale right

So the weights must be random — but *how big* should those random numbers be? This turns out to matter enormously, and it connects straight to the gradient-flow story from the last lesson.

Picture a signal passing forward through the layers, multiplied by the weights at each one. If the weights are too small, the signal shrinks a little at every layer, and after 20 layers it has faded to essentially zero — the network cannot tell its inputs apart, and since a dead forward signal means a dead backward signal too, nothing learns. If the weights are too large, the opposite happens: the signal grows until it blows up, or it slams activations like sigmoid and tanh into their flat "saturated" zones where their sensitivity drops to zero — and again the gradient dies. The weights need to be *just* the right size to keep the signal steady as it travels through the network.

The recipes: Xavier and He

Happily, the right size can be worked out exactly, and it depends on how many inputs feed into a layer. The idea is simple: pick the random scale so the signal comes out of each layer at about the same size it went in — no shrinking, no growing.

Xavier (Glorot) initialisation computes that scale for symmetric activations like tanh, and it was the fix that first let deep tanh networks train reliably.

He (Kaiming) initialisation adjusts it for ReLU. Because ReLU throws away all the negative values, it roughly *halves* the signal at every layer — so He simply doubles the variance to make up for the half that ReLU discards. Use He for ReLU networks and Xavier for tanh; use the wrong one on a deep network and it will silently fail to learn.

One small but famous exception: biases can safely start at zero (the random weights already break symmetry), *except* the LSTM's "forget gate" bias, which is usually set to 1. That nudges the gate to *remember* by default at the start, keeping the memory highway open long enough for the network to learn when it actually should forget.

The actual formulas, and fan-in vs fan-out

Worth carrying the numbers. Let fan_in be the number of inputs to a layer and fan_out the number of outputs. He/Kaiming (for ReLU) uses variance 2/fan_in. Xavier/Glorot (for tanh/sigmoid) uses either 1/fan_in or the symmetric 2/(fan_in + fan_out). Why the two versions of Xavier? Preserving the signal's variance on the forward pass wants 1/fan_in; preserving the gradient's variance on the backward pass wants 1/fan_out; averaging the two (2/(fan_in+fan_out)) is the compromise that keeps both roughly stable. He fixes on fan_in because for ReLU the forward-pass halving is the dominant effect to correct.

Uniform or normal?

Both Xavier and He come in a normal and a uniform flavour, and they're near-equivalent in practice. The normal version draws from N(0, variance). The uniform version draws from U(−limit, +limit) with the limit chosen to give the *same* variance (for Xavier uniform, limit = √(6/(fan_in+fan_out))). Frameworks default to one or the other; the difference rarely matters, but know that "Xavier uniform" and "Xavier normal" are the same idea with different sampling shapes.

Orthogonal initialisation for recurrence

For RNNs and very deep near-linear stacks, there's a better choice than random Gaussian: orthogonal initialisation, where the weight matrix is initialised to be orthogonal (its rows/columns are unit vectors at right angles). An orthogonal matrix has the property that it preserves vector norms under multiplication — so applying it repeatedly (as an RNN does across time steps) neither grows nor shrinks the signal. That's exactly the property you want when the *same* matrix is multiplied hundreds of times, which is why orthogonal init helps recurrent and deep-linear networks specifically.

Modern architectures soften the sensitivity — but don't remove it

Here's the honest caveat: "use the wrong init and the network silently fails" is true for a deep plain network, but much less so once you add residual connections and BatchNorm/LayerNorm. Normalisation re-centres and re-scales activations at every layer, which repairs a lot of a bad initial scale, and residual shortcuts keep gradients flowing regardless. So a ResNet or a normalised transformer is far more *forgiving* of initialisation than an old-style plain net. Init still matters — it affects early-training stability and final quality — but it's a smaller cliff than the unqualified claim suggests.

Transformer-specific initialisation

Very deep transformers need extra care beyond He/Xavier. Because each layer adds to the residual stream, naive init lets the residual-stream variance grow with depth, destabilising training — so large models scale the residual-branch weights down by a factor related to depth (e.g. 1/√(2N) schemes) to keep the stream stable. The embedding and output layers often get their own scaling, and the LayerNorm gain/bias start at 1/0. This is why deep transformers historically needed careful warmup — and why good residual-scaling schemes let them train more stably.

Diagnosing an init problem

You can catch a bad initialisation before wasting a training run. Check the loss at step zero: for a K-class classifier it should be ≈ ln(K) (e.g. ~2.3 for 10 classes) — a wildly different value means the output scale is off. Log the activation variance per layer on the first forward pass: healthy init keeps it roughly constant across layers; a steady decay or explosion means the scale is wrong. Also check per-layer gradient norms, the dead-ReLU count, and run NaN/Inf checks. These five-minute checks tell you the init is sane before you commit GPU hours.

Key points

• First rule: never initialise all weights to zero — break symmetry with randomness. If every neuron in a layer starts identical, they compute the same output, get the same gradient, and update in lockstep, so they stay identical forever — a whole layer collapses to the behaviour of one neuron. Random initial weights make neurons different from the start, which is the only way they can specialise into different features. Biases can safely start at zero, since the random weights already do the symmetry-breaking.
• Second rule: get the scale right, because it decides whether the signal survives. Too-small weights shrink the signal a little at each layer until, several layers deep, it has faded to zero and nothing learns. Too-large weights either blow the signal up or push tanh and sigmoid into their flat saturated zones where the gradient dies. The right scale keeps the signal about the same size from layer to layer — and it can be computed exactly from how many inputs each layer has.
• Match the recipe to the activation: Xavier for tanh, He for ReLU. Xavier (Glorot) picks the scale that preserves signal size through symmetric activations like tanh. He (Kaiming) adjusts it for ReLU: since ReLU discards the negative half of the signal, halving it each layer, He doubles the variance to compensate. Use the wrong one on a deep network — Xavier with ReLU, say — and the signal quietly decays layer by layer, and the network fails to train even though nothing looks obviously broken.
• One famous exception: the LSTM forget-gate bias starts at 1, not 0. A forget gate initialised at zero passes only about half the memory forward each step, so long-range information is erased before the model ever learns when to keep it. Setting the bias to 1 makes the gate lean toward "remember" by default, keeping the memory highway open at the start of training. Everywhere else, zero is a perfectly good default for biases.
• Carry the formulas and the specialised variants. He (ReLU): variance 2/fan_in. Xavier (tanh/sigmoid): 1/fan_in (preserves forward variance) or 2/(fan_in+fan_out) (compromise between forward and backward — fan_out preserves the gradient's variance). Uniform and normal flavours are equivalent (same variance, different sampling shape). For RNNs and deep near-linear stacks, orthogonal initialisation preserves vector norms under repeated multiplication, which is exactly what you want when the same matrix is applied many times.
• Modern architectures forgive init more, transformers need residual scaling, and you can diagnose it fast. "Wrong init = silent failure" holds for deep plain nets but is softened a lot by residual connections and BatchNorm/LayerNorm, which repair bad scale at every layer — init still matters for stability and final quality, just less of a cliff. Deep transformers scale residual-branch weights down with depth (so the residual stream doesn't grow) and this is why they needed warmup. Sanity-check init before a full run: loss at step 0 ≈ ln(K), roughly constant activation variance across layers, healthy per-layer gradient norms, low dead-ReLU count, and no NaN/Inf.

Recap

• Never initialize all weights to zero: every neuron in a layer then computes the same thing and receives the same gradient, so they stay identical forever and the layer has the capacity of one neuron — you must break symmetry with randomness.
• Scale decides survival: too-small weights fade the forward signal toward zero over depth (and vanish the backward gradient with it); too-large weights blow the signal up or saturate tanh/sigmoid into their flat regions. Init is gradient flow at step zero.
• Match the recipe to the activation: Xavier/Glorot for tanh/sigmoid, He/Kaiming for ReLU — He uses 2× the variance specifically to offset ReLU discarding the negative half of its inputs. Using Xavier for ReLU (or He for tanh) makes a deep net silently fail to learn.
• LSTM exception: initialize the forget-gate bias to 1 so the gate leans toward *remember* from the start, keeping the cell-state memory highway open; the other biases can be 0.
• Formulas: He = variance 2/fan_in; Xavier = 1/fan_in (forward) or 2/(fan_in+fan_out) (a forward+backward compromise); uniform and normal draws work about equally well.
• Orthogonal init for RNNs and deep linear stacks: an orthogonal matrix preserves vector norms under repeated multiplication, so the signal neither shrinks nor grows as it passes through many identical transforms.
• Residuals + norm forgive bad init a lot (they repair scale at every layer), so it's a soft failure not a cliff in modern nets; deep Transformers still scale residual-branch weights down with depth (why they needed warmup). Sanity-check init before a full run: loss at step 0 should be ≈ ln(K) for K classes.

Check your understanding