Dropout and Regularization: The Ensemble View

Dropout is described as "preventing overfitting by randomly turning off neurons." That is technically correct but misses the deeper picture. Dropout trains an exponential ensemble of architectures simultaneously and approximates their average at inference. Understanding this view explains why dropout works, when it does not, and what the alternatives accomplish.

Overfitting is the failure mode where a model memorises the training data rather than learning its structure. The solution is regularisation: constraining the model's capacity or the complexity of solutions it can find. L2 regularisation, L1 regularisation, and dropout are three different inductive biases, each with a geometric interpretation that reveals when to use which.

L2 regularisation: Gaussian prior on weights

L2 adds λ * ||w||^2 to the loss. The update becomes: w ← w - η * (∇L + 2λw) = w * (1 - 2ηλ) - η * ∇L. Each weight is decayed toward zero at every step, which is why L2 is also called weight decay. In Bayesian terms, L2 is equivalent to placing a Gaussian prior on the weights: it expresses the belief that weights should be small unless the data strongly justifies otherwise. Large weights are penalised quadratically, so a few very large weights are penalised more than many moderate ones. L2 encourages small, distributed weights.

L1 regularisation: Laplace prior and sparsity

L1 adds λ * ||w|| to the loss. The gradient is λ * sign(w) — a constant push toward zero regardless of weight magnitude. This drives small weights exactly to zero, producing sparse solutions. In Bayesian terms, L1 is a Laplace prior. It is used when you expect many features to be irrelevant and want the model to select a sparse subset. In deep learning L1 is less common than in linear models because neural network weights are harder to interpret as feature selectors.

Dropout: the ensemble interpretation

Dropout (Srivastava et al., 2014) randomly sets each neuron's activation to zero during training with probability p (typically 0.1–0.5). At each forward pass, a different random subset of neurons is active. With n neurons, there are 2^n possible architectures, each trained on a random subset of training examples. Dropout trains all of them simultaneously sharing weights.

At inference, all neurons are active and activations are multiplied by (1-p) — the expected fraction active during training. This approximates averaging the predictions of all 2^n sub-networks (geometric mean approximation). Ensemble methods consistently outperform single models; dropout makes this computationally free.

Why dropout prevents co-adaptation

Without dropout, neurons can co-adapt: neuron A learns to fix the errors of neuron B, and neither can function independently. A co-adapted group of neurons jointly memorises training patterns. With dropout, each neuron must learn features that are useful even when its co-adaptors are absent. This forces the network to learn more distributed, redundant representations — which generalise better.

Inverted dropout: keeping inference efficient

A subtle implementation detail. With standard dropout (scale at inference), you multiply all activations by (1-p) at test time — an extra operation at every inference call. Inverted dropout instead divides by (1-p) during training, scaling up active neurons to compensate for the ones dropped. Inference requires no scaling. All deep learning frameworks use inverted dropout by default.

When dropout underperforms

Dropout works best in large, overparameterised networks where co-adaptation is a real risk. It is less effective on: very small networks (not enough neurons to form co-adapted groups); convolutional layers (spatial correlation means dropping individual activations still leaves correlated neighbours active — SpatialDropout, which drops entire channels, works better); and Transformers, where attention already provides a form of regularisation and dropout is often set very low (p=0.1) or omitted in later layers.

For small datasets, the dominant regularisation tools are data augmentation (the most effective for vision), weight decay, and early stopping — not dropout.

Early stopping as regularisation

Stopping training before full convergence is a regularisation strategy. As a model trains past the point of minimum validation loss, it begins to memorise training noise. The optimal stopping point trades off training loss (lower = more memorised) against validation loss (lower = better generalised). Combined with a validation set and a patience parameter (stop if no improvement for k epochs), early stopping is often the most practical regularisation for limited-data regimes.

Try on Colab: train an overparameterised MLP on a small dataset (500 training examples of CIFAR-10). Train four variants: no regularisation, L2 weight decay, dropout (p=0.3), and both combined. Plot training loss and validation loss curves for all four. The gap between training and validation loss is the overfit signal — watch how each regulariser closes it.