Regularisation: The Geometric Picture of Why L1 Is Sparse and L2 Is Not

Most explanations of regularisation say "L1 produces sparsity, L2 does not." Almost none explain why. The answer is geometric: the L1 ball has corners that sit exactly on coordinate axes, so the constrained optimum tends to land on a corner — zeroing a weight. The L2 ball is a smooth sphere with no corners. This post makes the geometry rigorous and then covers every other form of regularisation you will encounter in practice.

Regularisation is the collection of techniques that prevent a model from fitting the training data too precisely — from memorising noise rather than learning signal. It is the primary tool for controlling variance in the bias-variance trade-off. Understanding it geometrically explains every form at once.

The constraint form vs the penalty form

The penalty form adds a regularisation term to the loss: min_w L(w) + λ Ω(w). The constraint form enforces a budget on the weights: min_w L(w) subject to Ω(w) ≤ t. These are mathematically equivalent by the KKT conditions of constrained optimisation — for every λ there exists a t that gives the same solution and vice versa. The constraint form has a cleaner geometric interpretation.

The geometry of L2 (Ridge)

The constraint Ω(w) = ||w||² ≤ t defines an L2 ball — a sphere in weight space centred at the origin. The unconstrained optimum w* (minimum of L(w) alone) is somewhere outside this ball if t is small. The constrained OLS optimum is where the loss contour ellipse first touches the L2 ball. Because the L2 ball is a smooth sphere, the touching point can be anywhere on the sphere's surface — including points with all weights non-zero. L2 regularisation shrinks all weights toward zero but does not zero any of them exactly. Ridge solution (for OLS): w_Ridge = (XᵀX + λI)⁻¹Xᵀy. The λI term makes the matrix invertible (solves multicollinearity) and shrinks eigenvalues of XᵀX uniformly.

The geometry of L1 (Lasso)

The constraint Ω(w) = ||w||₁ ≤ t defines an L1 ball — a diamond (in 2D) or cross-polytope (in higher dimensions). Critically: the L1 ball has corners, and those corners sit exactly on the coordinate axes (where one weight is non-zero and all others are zero). The loss contour ellipse, when it expands from the unconstrained optimum toward the origin, is geometrically most likely to first touch a corner of the L1 ball — because corners stick out. At a corner, all but one weight is exactly zero. This is the sparsity-inducing mechanism. It is a geometric inevitability, not a numerical quirk. Lasso has no closed form (the L1 norm is not differentiable at zero); it is solved with coordinate descent or ISTA/FISTA (proximal gradient methods).

Elastic Net: combining both

Elastic Net: min_w L(w) + λ₁||w||₁ + λ₂||w||². Combines the sparsity of L1 with the stability of L2 (Ridge handles correlated features; Lasso arbitrarily picks one). Useful when: you want feature selection but your features are correlated (Lasso picks one arbitrarily; Elastic Net keeps groups together).

Other forms of regularisation

Dropout (Srivastava et al., 2014): randomly set each neuron's activation to zero with probability p during training. At test time, multiply all activations by (1-p) to maintain expected values. Interpretation 1: prevents co-adaptation — neurons cannot rely on specific other neurons always being present. Interpretation 2: approximates Bayesian inference over an exponential number of thinned networks. Interpretation 3: ensemble view — different dropout masks produce different architectures; inference averages them. Early stopping: halt training when validation loss stops decreasing. Mathematically equivalent to L2 regularisation under certain conditions (Goodfellow et al.): a model trained for T gradient steps behaves similarly to a model with weight decay λ ≈ 1/(α T) where α is the learning rate. Weight decay: explicitly added to the gradient update: w ← w(1 - αλ) - α ∇L. For SGD, weight decay = L2 regularisation. For Adam, they diverge — weight decay applies to the parameter directly, not to the adaptive gradient, which is why AdamW (weight decay) is preferred over Adam+L2 for transformers. Data augmentation: artificially expand the training set with label-preserving transformations (flips, crops, colour jitter for images; back-translation for text). Reduces overfitting by increasing effective training set size. Label smoothing: replace hard one-hot targets with soft targets (1-ε for the true class, ε/(K-1) for others). Prevents the model from becoming overconfident. Equivalent to adding a KL divergence penalty between the model's output distribution and a uniform distribution.

Choosing the regularisation coefficient

λ is a hyperparameter. Too small: no regularisation, overfitting. Too large: underfitting (weights driven to zero). Select via cross-validation on the validation loss. For Lasso, the regularisation path (how the solution changes as λ varies from 0 to ∞) can be computed efficiently — features enter the model one by one as λ decreases, giving a feature ranking.

Interview questions on this topic

"Why does L1 regularisation produce sparsity but L2 doesn't? Explain geometrically." — The L1 ball (constraint region) is a diamond with corners on the coordinate axes. The loss function's contours tend to touch those corners first when they expand toward the origin, placing the solution at a corner where all but one weight is zero. The L2 ball is a sphere with no corners — the touching point can be anywhere, so no weight is forced to exactly zero.

"What is the difference between L2 regularisation and weight decay in the context of Adam?" — For SGD they are equivalent: L2 penalty adds λw to the gradient, weight decay multiplies parameters by (1-λ). For Adam, they are not equivalent: L2 adds λw to the gradient before the adaptive scaling, which means the effective penalty varies per-parameter. Weight decay (AdamW) applies the decay directly to parameters after the adaptive update, giving a consistent and better-calibrated penalty. AdamW trains better for large language models.

"How is dropout related to ensemble methods?" — Each dropout mask defines a different 'thinned' subnetwork. Training with dropout approximates averaging over all 2^H possible subnetworks (H = number of units). At test time, using the full network with scaled weights approximates the geometric mean of all these subnetworks' predictions — an implicit ensemble. This is why dropout improves generalisation.

"When would you prefer Elastic Net over Lasso?" — When features are correlated. Lasso tends to select one feature from a correlated group arbitrarily and zeroes the others. Elastic Net uses the L2 penalty to group correlated features together (they shrink together rather than one being selected). Also when p > n (more features than samples) — Lasso selects at most n features; Elastic Net can select more.

Try on Colab: fit Lasso, Ridge, and Elastic Net on the Diabetes dataset using LassoCV/RidgeCV/ElasticNetCV. Plot the coefficient paths as λ varies (regularisation path plot). Observe which features go to zero first in Lasso vs Ridge. Plot the number of non-zero coefficients vs λ. Verify that Elastic Net keeps more features than Lasso at the same regularisation strength when features are correlated.