Regularisation Geometry

L1 vs L2 geometry, Lasso sparsity, Ridge shrinkage, elastic net

Picture two students cramming for an exam. One learns the ideas — the reasoning behind each answer. The other memorises the practice answer key word for word. On that practice test the memoriser scores 100% and the thinker 90%. But on the real exam, with new questions, the thinker sails through and the memoriser falls apart — they learned the key, not the subject. That memoriser is a model that has overfit, and regularisation is how we stop it.

Here is where it bites in real life. You are predicting house prices with 100 features for only 200 houses — size, rooms, lot size, distance to school, and ninety-odd more, many of them overlapping. You fit plain least squares and the training fit looks incredible: R² of 0.98. Then you check houses it has never seen: R² of 0.51. It memorised the training set, noise and all.

To see exactly how the fix works, don't stay in the abstract — go back to the five houses from the linear regression module. There, with size (hundreds of sqft) as the only feature, OLS solved slope = Sxy/Sxx = 1900/250 = 7.6 in one step, with no penalty attached at all: plain least squares has no reason to hold back, and with one feature and five houses there isn't even room for it to misbehave. The trouble starts once you're fitting a hundred features to two hundred houses instead of one feature to five — now there is enormous spare freedom, and least squares will use every bit of it to bend through noise. Watch what happens to that same slope of 7.6 once a penalty is added below; the arithmetic is exact, not illustrative.

Why it overfits, and the one-line fix

Least squares has exactly one instruction: make the training error as small as possible, by any means. Nothing tells it to hold back. Give it 100 knobs to fit 200 points and it will use the spare freedom to bend through the noise — a giant positive weight here cancelled by a giant negative one there, contortions that fit these houses and no others.

So we change the instruction. We add a second piece to the loss: a penalty that grows with the size of the weights. The new loss is "training error plus lambda times the size of the weights," where lambda is a dial for how much we care.

Now watch the mechanism, because this is the whole thing. The model is trained by gradient descent, which does one job — nudge the weights, step by step, in whatever direction makes the loss smaller. The moment those big weights start adding a big number to the loss, gradient descent sees that cost and does the only thing it knows: it pushes the weights back down to bring the loss down again. Small weights are not a rule we impose from outside. They are what the model settles on by itself, once big weights start costing it. That is regularisation in one sentence: make big weights expensive, and gradient descent keeps them small.

Two ways to measure "size", two different results

There are two honest ways to measure how big the weights are, and the choice matters more than you would guess — and now we can measure exactly how much, reusing the five-house slope.

Square each weight and add them up — that is L2, also called Ridge. For one feature, mean-centered exactly the way OLS was derived, Ridge's own one-step formula is slope = Sxy / (Sxx + λ) — the same Sxy/Sxx from OLS, just with λ added into the denominator before dividing. At λ=0 that's 1900/250=7.6, the untouched OLS slope, as it must be. Push λ=250 (equal to Sxx itself) and the slope becomes 1900/(250+250) = 1900/500 = 3.8 — exactly half, because doubling the denominator exactly halves the ratio. Push harder, λ=1900: 1900/(250+1900) = 1900/2150 ≈ 0.884 — small, but never zero, no matter how large λ gets. That is the geometric picture made numeric: Ridge shrinks the weight smoothly toward zero, but Sxy/(Sxx+λ) can only approach zero in the limit, never land on it for any finite λ.

Add up the plain sizes instead (ignoring sign) — that is L1, also called Lasso. Lasso's one-step formula for a single centered feature is a soft threshold: slope = sign(Sxy) × max(|Sxy| − λ/2, 0) / Sxx — subtract λ/2 straight off the co-movement sum Sxy before dividing, and clamp at zero if that goes negative. At λ=0 this also reduces to 1900/250=7.6, matching OLS exactly. At λ=500: max(1900−250,0)/250 = 1650/250 = 6.6 — a smaller absolute drop than Ridge's proportional shrink at a comparable penalty. But push λ to 3800 (so λ/2=1900, exactly cancelling Sxy): max(1900−1900,0)/250 = 0/250 = 0 — exactly zero, no rounding. The feature is switched off completely, at a specific, finite, computable λ. Compare the two at their own natural stopping points: Ridge at λ=1900 still reports 0.884, a live if small weight; Lasso at λ=3800 reports a hard 0 — gone. That is the entire "diamond vs. circle" geometry from a picture, now as two numbers you can check with a calculator.

There is a plainer way to say why the two land differently. L1's derivative with respect to the weight is a constant ±λ — a steady force, independent of how small the weight already is — so it can walk the weight all the way to zero and stop. L2's derivative is 2λ×weight — a force that shrinks along with the weight itself — so as the weight gets small the pull gets weaker too. It slows down and stalls just short of zero, which is exactly the 0.884 that never quite becomes 0.

When to use which, and the trap

Use Lasso (L1) when you believe only a handful of features truly matter and you want the model to pick them out for you. Use Ridge (L2) when you think many features each add a little, or when features are correlated and you want to keep them together. (A blend called elastic net does a bit of both.)

And one trap falls straight out of "we penalise weight size": a feature measured in dollars needs a tiny weight, while a yes/no feature needs a big one. The same penalty hits them completely unequally — the big-scale feature barely feels it, the small-scale one gets hammered. So standardise your features first (put them all on the same scale), or the penalty is quietly punishing features for their units instead of judging how useful they are.

Correlated features made concrete: the duplicate-column trap

Watch the "features clearly work together" case turn into hard numbers too. Suppose the five-house data gets a duplicate column by accident — someone adds size again, unchanged, as a second feature. Now the model is

$weight₁×size + weight₂×size = (weight₁+weight₂)×size$

Only the *sum* of the two weights affects any prediction, so plain OLS has infinitely many equally good answers:

$weight₁=7.6, weight₂=0 or weight₁=3.8, weight₂=3.8 or weight₁=−100, weight₂=107.6$

all fit the five houses identically. Least squares has no preference among them; whichever numerical solver you run will simply return one of the infinitely many ties, and a slightly different solver, or a slightly different batch of houses, can return a wildly different split — exactly the "weights wobble but predictions stay fine" warning from the collinearity section of the linear regression module.

Ridge breaks the tie in a specific, checkable way: among every (weight₁, weight₂) pair summing to 7.6, it additionally picks the one that minimises weight₁²+weight₂² — and by symmetry that is the even split, weight₁=weight₂=3.8. (Write weight₁=3.8+d, weight₂=3.8−d: the sum of squares is 2×(3.8²+d²), minimised exactly at d=0.) As λ→0⁺, Ridge doesn't merely shrink the duplicated weight — it deterministically lands on the *minimum-norm* solution among all the tied OLS answers, and for two identical columns that minimum-norm answer is always the even split.

Lasso ties in a different way. Since both weights are non-negative here, |weight₁|+|weight₂| = weight₁+weight₂ = 7.6 for every single point on that same tied line — the L1 penalty is exactly the same number, 7.6×λ, no matter how the 7.6 is divided between the two columns. So Lasso's objective genuinely cannot prefer one split over another either — but unlike Ridge's bowl-shaped penalty, its penalty is flat along the whole tied line, so there is no unique minimum to fall back on. In practice a solver (coordinate descent) breaks that flat tie arbitrarily, typically landing on a corner: weight₁=7.6, weight₂=0, or the reverse, whichever column happens to get updated first. That is the mechanism behind "which correlated feature Lasso keeps can flip from one run to the next" — it isn't a bug, it's what an unpenalised flat direction in the loss does to any greedy solver.

The real tradeoff underneath: bias for variance

Why does shrinking weights help at all? Because it trades one kind of error for another. An unconstrained model has *low bias* (it can fit any shape) but *high variance* (it swings wildly from one training sample to the next — that's the overfitting). Adding a penalty deliberately introduces a little bias — the weights are pulled away from the perfect training fit — in exchange for a large drop in variance. Ridge's own 3.8 at λ=250, versus OLS's 7.6, makes the bias concrete: a gap of exactly 3.8 has been deliberately introduced on this one example. Five hand-picked houses can't show the variance side of the trade (that needs repeated resampling), but it is the entire justification: on a genuinely noisy, high-dimensional dataset, that same shrinkage is what keeps the fitted weight from swinging wildly between training runs. The goal is never "small weights for their own sake"; it's *lower error on unseen data*. You accept some bias because the variance you kill is worth more. That framing — regularisation buys variance reduction at the price of bias — is the one interviewers want to hear.

How you actually pick lambda

Lambda isn't guessed; it's *tuned*. Neither 250 nor 3800 above was chosen by looking at the five houses' own training error — training error would tell you to prefer λ=0 every time, since that's where it's smallest. Instead you sweep a range of values and, for each, measure error on held-out data with cross-validation — a validation curve of error versus lambda. Too little penalty and both train and validation error show the overfit gap; too much and the model underfits and both climb. The sweet spot is the lambda that minimises validation error. scikit-learn ships this as `RidgeCV` and `LassoCV` so the search is built in. Never pick lambda by looking at training error — it always prefers zero penalty.

Ridge has a one-step formula too — and it explains why it helps

Just like OLS, Ridge has a closed form: $θ̂ = (XᵀX + λI)⁻¹Xᵀy$. That single-feature formula, slope = Sxy/(Sxx+λ), computed above is the p=1 special case of this same matrix formula — for one feature, XᵀX is just Sxx (a 1×1 matrix), and adding λI means adding λ to that single number, exactly the +λ that shrank 7.6 down to 3.8. Notice the only change from OLS is the $+λI$ added to the diagonal before inverting. That's not cosmetic — when features are correlated, XᵀX is nearly singular and blows up on inversion (the exact source of those wild, unstable weights). The duplicate-column example above is the extreme case of this in miniature: two identical columns make XᵀX exactly singular — literally uninvertible — which is why plain OLS has infinitely many tied solutions there. Adding $λI$ lifts the diagonal and makes the matrix cleanly invertible again, and as shown above it picks out the even split as that restored solution. So Ridge literally *stabilises the inversion*, which is why it tames collinearity.

The naming mess across libraries

This trips people up constantly, so nail it. In scikit-learn, `Ridge` and `Lasso` take alpha as the penalty strength (bigger alpha = more regularisation). But `LogisticRegression` and `LinearSVC` take C, which is the *inverse* (C = 1/λ, so *smaller* C = more regularisation). And `ElasticNet` takes alpha for overall strength plus l1_ratio to mix L1 and L2 (l1_ratio=1 is pure Lasso, 0 is pure Ridge). Same idea, three different dials.

Don't penalise the intercept

One subtlety: the intercept (bias term) is usually *not* regularised. Penalising it would drag your predictions toward zero for no good reason — the intercept just anchors the overall level, it isn't a feature whose influence you want to shrink. Libraries handle this for you, but it's why you *center* features (and why standardising matters): with centered features the intercept stays meaningful and the penalty only touches the actual feature weights.

Lasso's sharper limits

Beyond the "which correlated feature gets kept is unstable" issue, Lasso has a hard structural limit: in a wide problem with more features than samples (p > n), it can select at most about n features before it runs out — a real problem in genomics or text where p is huge. It can also over- or under-select depending on lambda. Elastic net was invented partly to fix exactly these Lasso failures: it keeps L1's sparsity while L2's presence lets it select more than n features and hold correlated groups together.

Not just linear regression

Finally, regularisation isn't a linear-regression trick — it's everywhere. It's the `C` in logistic regression and SVMs, the margin-softening in SVMs, and weight decay in neural networks (L2 on the network's weights). The penalty interacts differently with each loss and solver, but the core move is identical: add a cost on complexity so the optimiser stops chasing the training noise.

Key points

• What regularisation does: it adds a penalty on weight size to the loss, so gradient descent keeps the weights small and the model simple. Reach for it any time you have many features relative to your data, or you see a big gap between training and test performance — the classic sign of overfitting. On the five-house example, plain OLS finds slope = Sxy/Sxx = 1900/250 = 7.6 with no penalty at all; Ridge's slope = Sxy/(Sxx+λ) pulls that down to 3.8 at λ=250. Use Ridge (L2) as your default; it shrinks everything gently and handles correlated features well. Use Lasso (L1) when you suspect most features are useless and you want the model to zero them out and hand you a short list. The dial is lambda (in scikit-learn often called alpha, or C = 1/lambda for logistic regression): more penalty means a simpler model.
• The trap: Lasso's feature picks get shaky when features are correlated. If two features carry nearly the same information, Lasso keeps one and zeros the other — but which one it keeps can flip from one training run to the next. Duplicate the five houses' size column exactly and OLS has infinitely many tied (weight₁,weight₂) splits summing to 7.6; Ridge deterministically settles on the even split (3.8, 3.8), the minimum-norm tie-break, while Lasso's flat penalty along that same tied line has no unique minimum, so a solver arbitrarily lands on a corner like (7.6, 0). So do not read Lasso's chosen features as gospel. If the selected set changes across cross-validation folds, switch to elastic net, which blends in a little Ridge and tends to keep correlated features together instead of picking one at random.
• The habit that is not optional: standardise your features before any regularised model. Because the penalty judges weights purely by size, a feature on a huge scale (income in dollars) needs a tiny weight and barely gets penalised, while a 0/1 flag needs a big weight and gets hammered — even if they are equally useful. Put every feature on the same scale first (subtract the mean, divide by the standard deviation). Skip this and the penalty punishes features for their units, not their usefulness, and the whole model tilts toward the large-scale ones.
• The framing to state out loud: regularisation trades a little bias for a big drop in variance, and lambda is tuned, not guessed. An unconstrained model is low-bias but high-variance (it overfits); the penalty adds bias to kill variance, and the target is lower error on unseen data, not small weights for their own sake. On the five houses, Ridge's slope of 3.8 versus OLS's 7.6 is exactly that bias, 3.8 worth, deliberately introduced — bought in exchange for a variance reduction that only shows up on noisier, higher-dimensional data. Pick lambda by cross-validation — sweep values and take the one that minimises validation error (`RidgeCV`/`LassoCV`) — never by training error, which always wants zero penalty. Too little penalty overfits; too much underfits; both raise validation error.
• Ridge has a closed form that shows why it works, and the library naming is a minefield. Ridge solves $θ̂ = (XᵀX + λI)⁻¹Xᵀy$; for one feature this collapses to slope = Sxy/(Sxx+λ) = 1900/(250+λ), reaching 3.8 at λ=250 and ≈0.884 at λ=1900, but never exactly 0. The $+λI$ lifts the diagonal so a near-singular XᵀX (from correlated or duplicated features) becomes cleanly invertible — that's literally how Ridge stabilises collinearity. On naming: scikit-learn's `Ridge`/`Lasso` use `alpha` (bigger = more penalty), `LogisticRegression`/`LinearSVC` use `C` = 1/λ (smaller = more penalty), and `ElasticNet` uses `alpha` plus `l1_ratio` to blend L1 and L2. Also: don't regularise the intercept — center features so it stays meaningful.
• Know Lasso's hard limits and that regularisation reaches far beyond linear regression. With more features than samples (p > n), Lasso can select at most about n features and its picks are unstable under correlation — elastic net was designed to fix both by keeping L1 sparsity while L2 lets it exceed n features and hold correlated groups together. Lasso's own soft-threshold, slope = sign(Sxy)×max(|Sxy|−λ/2,0)/Sxx, hits exactly 0 at λ=3800 on the five-house example — the hard-zero behaviour Ridge structurally can't match. And the same idea is everywhere: the `C` in logistic regression and SVMs, the soft margin in SVMs, and weight decay (L2) in neural networks — add a cost on complexity so the optimiser stops chasing training noise.

Recap

• Regularisation = penalty on weight size added to the loss. Gradient descent chases smaller loss → weights stay small, model stays simple. Five-house example: OLS slope = Sxy/Sxx = 1900/250 = 7.6, unpenalised.
• L2 (Ridge): slope = Sxy/(Sxx+λ) → 3.8 at λ=250, ≈0.884 at λ=1900 — shrinks smoothly, never exactly zero. L1 (Lasso): slope = sign(Sxy)×max(|Sxy|−λ/2,0)/Sxx → 6.6 at λ=500, exactly 0 at λ=3800 — feature selection.
• Always standardise first — the penalty judges weights by size, not usefulness.
• Duplicate/correlated feature (weight₁+weight₂=7.6, tied): Ridge deterministically splits evenly (3.8, 3.8 — minimum-norm tie-break); Lasso's flat penalty along the tied line has no unique minimum, so a solver arbitrarily lands on a corner (7.6, 0) — why Lasso's picks flip across runs.
• Trades a little bias for a big drop in variance; Ridge's 3.8 vs OLS's 7.6 on the five houses is exactly that bias, 3.8 worth. λ is tuned via cross-validation, not guessed.
• Ridge has a closed form $θ̂=(XᵀX+λI)⁻¹Xᵀy$, reducing to Sxy/(Sxx+λ) for one feature — λI restores invertibility exactly where duplicate/correlated columns make XᵀX singular. Library naming (C = 1/λ) is a minefield.

Check your understanding