PCA from First Principles
Covariance matrix, explained variance, when PCA fails
You are building a face recognition system. Each image is 100×100 pixels — 10,000 numbers per image. Training a model on raw 10,000-dimensional inputs is slow and prone to overfitting. But most of those dimensions carry redundant information: neighboring pixels are highly correlated, and faces share common structure — eyes roughly here, nose here, mouth here.
The naive fix is to drop some features. But which ones? Dropping pixel 4,512 and keeping pixel 4,513 is arbitrary — both carry similar information. What you want is to find the *directions* where faces actually vary, and represent each face as its coordinates along those directions.
PCA does exactly this. Centre the data (subtract the mean face). Compute the covariance matrix $Sigma = X^T X / (n-1)$ — a 10,000 × 10,000 matrix encoding how every pixel correlates with every other pixel. Find the eigenvectors of this matrix. The eigenvector with the largest eigenvalue is the direction along which face images vary most. Project every image onto the top $k$ eigenvectors. You have compressed 10,000 dimensions down to $k$ — say 50 — while retaining whatever fraction of variance those 50 directions explain. A scree plot of eigenvalues sorted in descending order shows the "elbow" where additional components stop explaining much variance.
Two things make PCA fail. First, scale: a feature measured in cents has 10,000× the variance of the same feature measured in dollars — converting dollars to cents multiplies every number by 100, and variance scales with the square of that factor (100² = 10,000×). PCA will identify "cents direction" as the first principal component — not because it contains more signal, but because its numbers are larger. Always standardise (z-score) before running PCA unless features share a natural common scale.
NOT this. Most people think "PCA removes correlated features." PCA does not select a subset of original features. It creates entirely new features that are linear combinations of *all* original features. The new features — principal components — are uncorrelated with each other by construction. But each PC mixes every original feature together. You cannot look at a principal component and say "this is pixel 4,512." The confusion matters because PCA cannot be used for feature selection if you need interpretability in the original feature space.
Key points
- Use PCA to reduce compute and overfitting, not to select features. When downstream models are slow or overfit on high-dimensional inputs, project to the top $k$ PCs first. Choose $k$ to retain 95% of variance, or use the scree plot elbow. Computing PCA via SVD of the data matrix $X$ is numerically superior to eigendecomposition of $X^T X$ — it avoids squaring the condition number.
- The production trap: scale sensitivity. A pixel modeled as Uniform[0, 255] has variance $255^2/12 approx 5{,}419$; a binary indicator at its own maximum variance (Bernoulli(0.5)) has variance $0.25$ — a ratio of roughly 21,700×. PCA will identify the pixel direction as the first principal component — not because it is more informative, but because it is numerically larger. Always z-score before PCA unless all features share a natural common unit. After PCA, whitening (dividing each PC score by $\sqrt{\lambda_k}$) gives unit variance in every direction and is required before k-means or GMMs.
- The diagnostic: compare PCA-compressed model accuracy against label-supervised baselines. If 95% of variance explains only 70% of classification accuracy, the remaining 5% likely contains the discriminative signal. Try LDA (Linear Discriminant Analysis) — it maximises between-class variance rather than total variance, and will find directions PCA discards when classes differ in low-variance directions. If PCA's reconstruction quality is high but downstream task quality is poor, switch to a supervised reduction method.
PCA keeps high-variance directions and discards low-variance ones. Always verify that the discarded variance does not contain the label signal — the information your classifier needs most may live exactly in the directions PCA throws away.
Recap
- PCA finds directions of max variance, not a subset of features — each PC mixes all original features.
- Recipe: center data → covariance Σ = XᵀX/(n−1) → eigenvectors → project onto top-k.
- PCs are uncorrelated by construction; eigenvalue ratio = fraction of variance explained per direction.
- Scale sensitivity is the trap: switching a feature from dollars to cents inflates its variance 10,000× (numbers ×100, variance ×100²); always z-score first.
- PCA is not feature selection — you can't say a PC "is pixel 4,512", so it loses original-space interpretability.
- 95% variance ≠ 95% accuracy: the label signal may live in the low-variance directions PCA discards.
- If PCA hurts the downstream task, use a supervised reduction (LDA/PLS) that maximises between-class variance.
Check your understanding
Q1. You have a dataset with covariance matrix Σ. Describe the PCA algorithm as an eigendecomposition problem, and explain what the principal components represent.
- A) PCA solves the linear system Σw = 0 to find directions of zero variance — these are the null space vectors of Σ. Principal components are the directions orthogonal to the null space, i.e., the column space of Σ. The explained variance for component k is tr(Σ) − λₖ, measuring how much variance remains after removing that direction from the total spread, which is always non-negative for PSD Σ.
- B) PCA finds orthonormal directions w₁,...,wₖ in order of maximum variance: w₁ = argmax_{‖w‖=1} wᵀΣw. This is the eigenvector problem Σw=λw — w₁ is the eigenvector of Σ with the largest eigenvalue λ₁; w₂ is the eigenvector with second-largest eigenvalue, subject to w₂⊥w₁. Geometrically, Σ's eigenvectors are the principal axes of the covariance ellipsoid.
- C) PCA computes the gradient ∇_w wᵀΣw = 2Σw and follows it to convergence. This is the power iteration algorithm: starting from random w, repeatedly compute Σw and normalise. The algorithm converges to the largest eigenvector. Subsequent components are found by deflation: subtract the rank-1 contribution of the found component and repeat.
- D) PCA minimises the reconstruction error ‖X − XWWᵀ‖²_F over orthonormal W. Setting the gradient to zero gives ΣW = WΛ where Λ is diagonal. This is equivalent to finding the eigenvectors of Σ but via an optimisation perspective rather than the spectral perspective. The principal components are the columns of W that minimise reconstruction error.
Q2. PCA on a dataset with 1000 features gives first two PCs explaining 95% of variance. A colleague uses these 2 PCs as features for a random forest. Which TWO of the following correctly describe what might go wrong?
- A) The random forest will be slower than on the original 1000 features because PCA produces dense features (every original feature contributes to each PC), making tree splits computationally expensive to evaluate. Sparse original features allow faster tree construction via efficient split-point search algorithms that skip zero entries.
- B) A forest built on just 2 PCs has far less room to find nonlinear interactions than one with access to all 1000 original features: even if those 2 components capture 95% of variance, every tree can only split on 2 axes, so any decision boundary that genuinely needs a 3rd discriminative direction becomes impossible for the forest to represent.
- C) PCA produces correlated features when only 2 PCs are used, because the first 2 PCs both contain information from the same underlying features. Random forests are sensitive to feature correlation — the variable importance scores become unreliable, and correlated PCs cause the forest to double-count certain directions during split selection.
- D) The 95% variance explained is for reconstruction, not prediction. The remaining 5% may be disproportionately predictive: if the label correlates with a low-variance direction, PCA discards exactly that direction, even though it is critical for the classifier — variance and label-relevance are different axes. Supervised reduction (LDA, PLS) preserves label-correlated dimensions instead.
Q3. Why must you subtract the mean before applying PCA? What goes wrong if you do not?
- A) PCA finds directions of maximum variance, measured relative to the mean: Var(X)=E[(X−μ)(X−μ)ᵀ]. Without centering, Σ_wrong=E[XXᵀ] is the second moment matrix — its first principal component is dominated by the mean direction, not the true spread. If the mean is large, the first PC is essentially just the mean direction, regardless of the actual variance structure.
- B) Mean subtraction has no effect on the eigenvectors of the covariance matrix — it only shifts the eigenvalues by a constant. Without centering, the eigenvalues of E[XXᵀ] are equal to those of E[(X−μ)(X−μ)ᵀ] plus ‖μ‖², so the PCs are identical but the explained variances are inflated. Centering is optional and is only necessary when you want accurate explained variance percentages.
- C) Without mean subtraction, PCA on E[XXᵀ] finds the correct covariance structure as long as the data is standardised (zero mean forced by normalisation). The mean-subtraction requirement is specific to sklearn's implementation choice; mathematically, you only need E[XXᵀ] to be positive definite, which holds when the data spans all dimensions.
- D) Mean subtraction converts PCA from an unsupervised to a supervised method: centering by the class-conditional mean, as in Fisher's LDA, introduces label information into the computation. Without centering by the global mean, PCA remains purely unsupervised. Subtracting the mean biases the first PC toward the decision boundary between classes rather than toward the true direction of maximum marginal variance in the feature space.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →