Neural Network Initialisation: He, Xavier, and Why the Wrong Init Kills Training

Why not initialise all weights to zero? Why not initialise them to large random values? The answer is signal variance: if every layer scales activations by more than 1, signals explode; if less than 1, signals vanish. Xavier and He initialisation are exact solutions to keeping variance constant through the network — one for linear activations, one for ReLU. This post derives them.

Initialisation seems like a boring implementation detail. It is not. Poor initialisation causes training to fail in the first few steps, before the data has had any effect. The right initialisation keeps the signal well-conditioned through the entire forward and backward pass from step one.

Why not all zeros?

If all weights are zero, all neurons in each layer compute the same output — they are symmetric. During backpropagation, they receive identical gradient updates and remain symmetric forever. The network behaves like a single neuron regardless of width. This is the symmetry breaking problem: random initialisation breaks symmetry so different neurons can learn different features.

Why not large random values?

Consider a linear network with L layers, each of width n, with weights initialised from N(0, σ²). The output of layer l is z_l = W_l z_{l-1}. The variance of z_l = n σ² × Var(z_{l-1}). After L layers: Var(z_L) = (nσ²)^L × Var(z_0). If nσ² > 1: variance explodes exponentially with depth. Activations saturate for sigmoid/tanh; gradients vanish. If nσ² < 1: variance shrinks exponentially with depth. Activations all collapse to zero; gradients vanish. We need nσ² ≈ 1, i.e., σ² = 1/n — the key insight.

Xavier / Glorot initialisation (linear / tanh activations)

Glorot & Bengio (2010) analysed both the forward pass (keep activation variance constant) and the backward pass (keep gradient variance constant). For a layer with n_in inputs and n_out outputs, the two conditions give different σ²: forward only: σ² = 1/n_in. Backward only: σ² = 1/n_out. Xavier/Glorot compromises: σ² = 2/(n_in + n_out). Equivalently, uniform initialisation: Uniform(-√(6/(n_in + n_out)), √(6/(n_in + n_out))). This is optimal for linear activations and approximately correct for tanh (which is approximately linear near zero). In PyTorch: torch.nn.init.xavier_uniform_(layer.weight).

He / Kaiming initialisation (ReLU activations)

ReLU(z) = max(0, z) kills half of the activations (all z < 0 become 0). This means only half of the neurons are active, halving the effective fan-in. He et al. (2015) derive: σ² = 2/n_in. Uniform version: Uniform(-√(6/n_in), √(6/n_in)). The factor of 2 compensates for ReLU zeroing half the values. For leaky ReLU with slope a: σ² = 2 / ((1+a²) × n_in). In PyTorch: torch.nn.init.kaiming_normal_(layer.weight, mode='fan_in', nonlinearity='relu'). He initialisation is the default for any network with ReLU activations (includes ResNets, VGG, most modern architectures).

LeCun initialisation (SELU activations)

Klambauer et al. (2017) derived the Self-Normalising Neural Network using SELU activations, which maintain mean 0 and variance 1 through layers without batch norm. LeCun initialisation: σ² = 1/n_in (the simplest case). When combined with SELU, this produces a self-normalising property that makes very deep networks trainable without batch norm.

Orthogonal initialisation (RNNs)

For recurrent networks, the hidden-to-hidden weight matrix W_hh is multiplied at every timestep: h_T = W_hh^T h_0. If the eigenvalues of W_hh are not exactly 1, the signal either explodes or vanishes over T steps. Orthogonal matrices have all eigenvalues on the unit circle — perfect for recurrent weights. Orthogonal initialisation: draw a random matrix, compute its QR decomposition or SVD, use the Q matrix. In PyTorch: torch.nn.init.orthogonal_(layer.weight).

Batch normalisation as a substitute

Batch normalisation (Post 58) normalises activations to have zero mean and unit variance within each mini-batch. This makes the network approximately insensitive to initialisation — even poor initialisations converge because batch norm continuously reconditions the signal. However, batch norm adds its own complexity (train vs test behaviour, batch size sensitivity) and is not always appropriate (small batch sizes, sequence models prefer layer norm).

Interview questions on this topic

"Why does He initialisation use 2/n_in while Xavier uses 2/(n_in + n_out)?" — ReLU kills half the input signal (zeros the negative half), so the effective fan-in is halved. Multiplying by 2 compensates. Xavier targets symmetric activations (linear, tanh) where the full fan-in contributes; the factor accounts for both forward and backward pass by averaging n_in and n_out.

"What happens if you initialise a very deep network with σ = 0.1?" — If σ = 0.1 and n = 1000 (layer width), then nσ² = 10 — signal will explode. If σ = 0.001 and n = 1000, then nσ² = 1 — might be stable. But the exact condition depends on the activation function. Without proper initialisation, gradients will explode or vanish within a few forward passes, and training will stall immediately.

"Can you train a network without random initialisation?" — Not with all-zero or all-constant initialisation due to symmetry breaking. However, structured initialisations exist: identity matrix initialisation for recurrent nets, pre-trained weights for transfer learning, or very specific deterministic constructions in theory. In practice, random initialisation with the correct variance is always used.

"What does the condition 'keep variance constant through layers' mathematically require?" — For a linear layer z = Wx with i.i.d. inputs xᵢ and i.i.d. weights W_{ij} ∼ N(0, σ²), Var(z_j) = n_in × σ² × Var(xᵢ). Setting this equal to Var(xᵢ) requires σ² = 1/n_in. This is the forward-pass condition. The backward-pass condition requires σ² = 1/n_out. Xavier/Glorot averages them.

Try on Colab: train a 20-layer MLP on MNIST with three initialisations: (1) all zeros, (2) N(0, 1), (3) He initialisation. Plot the distribution of activations at layer 10 for each case after the first forward pass. Show that zeros give all-zero activations (dead network), N(0,1) gives exploding activations, and He gives well-conditioned activations. Then plot training loss over 10 epochs for each — only He trains successfully.