OLS and Linear Regression: Normal Equations, Gauss-Markov, and When It Breaks

Linear regression is the most-used model in all of statistics. It is also the most misunderstood — practitioners use it without knowing what OLS actually minimises, why the normal equations work geometrically, or what the Gauss-Markov theorem guarantees. This post derives it from scratch and maps exactly where each assumption can break in real data.

Ordinary Least Squares (OLS) linear regression is the baseline against which every other model is judged. If your problem is not substantially non-linear, OLS with proper feature engineering often matches or beats complex models while remaining fully interpretable. Understanding it deeply means understanding projection geometry, the normal equations, and the five Gauss-Markov assumptions.

The setup

Given n observations and d features: design matrix X ∈ ℝⁿˣᵈ (each row is one sample), target vector y ∈ ℝⁿ. We want to find weights w ∈ ℝᵈ such that ŷ = Xw ≈ y. OLS minimises the residual sum of squares: RSS(w) = ||y - Xw||² = (y - Xw)ᵀ(y - Xw) = yᵀy - 2wᵀXᵀy + wᵀXᵀXw.

Deriving the normal equations

Take the gradient of RSS with respect to w and set to zero: ∂RSS/∂w = -2Xᵀy + 2XᵀXw = 0. Therefore: XᵀXw = Xᵀy. This is the normal equations. If XᵀX is invertible (X has full column rank): w_OLS = (XᵀX)⁻¹Xᵀy. The matrix (XᵀX)⁻¹Xᵀ is called the Moore-Penrose pseudoinverse of X, often written X⁺.

Geometric interpretation: projection onto the column space

The columns of X span a d-dimensional subspace of ℝⁿ — the column space col(X). The vector Xw is a linear combination of X's columns. OLS finds the w such that Xw is the orthogonal projection of y onto col(X). Why orthogonal? Because the residual e = y - Xw must be perpendicular to every column of X: Xᵀe = Xᵀ(y - Xw) = 0 — exactly the normal equations. Geometrically, OLS drops a perpendicular from y to the nearest point in the subspace col(X). This is the minimum-distance (minimum RSS) solution.

The Gauss-Markov theorem: when OLS is optimal

Under five assumptions, OLS is the Best Linear Unbiased Estimator (BLUE) — no other linear unbiased estimator has lower variance: (1) Linearity: y = Xw + ε (the relationship is truly linear). (2) Strict exogeneity: E[ε|X] = 0 (errors are mean-zero given the features; no omitted variable bias). (3) No perfect multicollinearity: X has full column rank (XᵀX is invertible). (4) Homoscedasticity: Var(εᵢ|X) = σ² for all i (constant error variance). (5) No autocorrelation: Cov(εᵢ, εⱼ|X) = 0 for i ≠ j (errors are uncorrelated across observations). Under (1)-(5), the OLS estimator is unbiased: E[w_OLS] = w*, and has variance Var(w_OLS) = σ²(XᵀX)⁻¹ — the smallest possible for any linear unbiased estimator.

R² and adjusted R²

Total Sum of Squares: TSS = Σᵢ(yᵢ - ȳ)² (total variance in y). Residual Sum of Squares: RSS = Σᵢ(yᵢ - ŷᵢ)². R² = 1 - RSS/TSS. Proportion of variance explained. R² always increases when you add features, even noise features — it cannot decrease. Adjusted R² penalises for the number of features: R²_adj = 1 - (RSS/(n-d-1)) / (TSS/(n-1)). Can decrease when you add useless features. R² = 0 for a model that predicts the mean everywhere; R² = 1 for a perfect fit. R² < 0 is possible if the model is worse than just predicting the mean.

When OLS breaks and what to do

Multicollinearity (assumption 3 violated): XᵀX becomes near-singular. OLS weights are numerically unstable with huge variance — small changes in data cause wild changes in weights. Detection: Variance Inflation Factor VIF_j = 1/(1-R²_j) where R²_j is the R² from regressing feature j on all other features. VIF > 10 is problematic. Fix: Ridge regression (adds λI to XᵀX → w_Ridge = (XᵀX + λI)⁻¹Xᵀy, always invertible) or drop correlated features. Heteroscedasticity (assumption 4 violated): error variance depends on features (common in financial data, count data). OLS is still unbiased but no longer BLUE — standard errors are wrong, inference is invalid. Fix: Weighted Least Squares (WLS) where each observation is weighted by 1/σᵢ², or robust standard errors (HC3 sandwich estimator). Autocorrelation (assumption 5 violated): common in time series (residuals at t are correlated with residuals at t-1). OLS is still unbiased but standard errors are wrong. Fix: add lagged features, use GLS (Generalized Least Squares), or switch to a time series model. Non-linearity (assumption 1 violated): add polynomial features, interaction terms, or use a non-linear model.

Interview questions on this topic

"Derive the OLS estimator. What does it mean geometrically?" — Take ∂RSS/∂w = 0 → XᵀXw = Xᵀy → w = (XᵀX)⁻¹Xᵀy. Geometrically: ŷ = Xw is the orthogonal projection of y onto the column space of X. The residual y - ŷ is perpendicular to all columns of X.

"What is R² and why can it be misleading?" — Proportion of variance explained, 1 - RSS/TSS. Misleading because: (1) always increases with more features even if they are noise, (2) high R² does not imply the model is correct — a misspecified model can have high R², (3) R² says nothing about predictive accuracy on new data.

"What happens to OLS when two features are perfectly correlated?" — XᵀX is singular (non-invertible); the normal equations have infinitely many solutions. Any combination of the two correlated weights that produces the same fitted values has identical RSS. In practice near-collinearity makes the inverse ill-conditioned — weights have huge variance. Ridge regression fixes this by adding λI.

"When would you use WLS instead of OLS?" — When the error variance is not constant (heteroscedasticity). For example: modelling average income by region (larger regions have lower variance in averages due to averaging over more people); fitting log-transformed data back on original scale; any setting where you know the precision of each observation varies. WLS weights each observation by 1/σᵢ² to give more influence to precise observations.

Try on Colab: fit OLS on the California Housing dataset. Plot residuals vs fitted values — if the plot shows a funnel shape, you have heteroscedasticity. Compute VIF for all features. Introduce artificial collinearity by adding a feature = 2×feature_1 + noise(0, 0.001). Show the VIF spike and the variance explosion in the OLS coefficient. Apply Ridge regression and show the coefficient stabilises.