PCA from Scratch: What the Eigenvectors Are Actually Capturing

PCA is described as "finding directions of maximum variance." That is correct but incomplete. The eigenvectors of the covariance matrix are orthogonal directions that capture as much of the data's variability as possible in order. The first principal component is the line closest to all data points. The eigenvalue tells you exactly how much variance each direction captures. This makes PCA's limitations as clear as its strengths.

Principal Component Analysis is one of the most universally used tools in data science — feature reduction, visualization, noise removal, anomaly detection — and one of the most commonly misunderstood. The mechanics are clear; the geometry is what makes it powerful.

The setup: what PCA is finding

You have n data points in d-dimensional space (d features, possibly correlated). PCA finds a new coordinate system where the axes are ordered by the amount of variance they explain. The first axis (PC1) is the direction in which the data varies most. The second axis (PC2) is perpendicular to PC1 and explains the most remaining variance. And so on.

Formally: find orthonormal vectors v_1, v_2, ..., v_k that maximise the variance of the projected data Var(X * v_i), subject to v_i ⊥ v_j for i ≠ j.

The covariance matrix and eigendecomposition

Center the data: X_c = X - mean(X). Compute the d×d covariance matrix: Σ = (1/n) X_c^T X_c. The principal components are the eigenvectors of Σ. The corresponding eigenvalues λ_i give the variance explained by each component: proportion of total variance = λ_i / Σ λ_j. The eigenvector with the largest eigenvalue is PC1; the second-largest gives PC2; and so on.

Why eigenvectors? Because the eigenvector equation Σv = λv says: the covariance matrix applied to v just scales it by λ. That means v is a direction in which the covariance matrix acts as a pure scalar — the most "natural" directions for the data's spread.

Computing PCA: eigendecomposition vs SVD

For d features, eigendecompose the d×d covariance matrix. For n < d (more features than samples), it is more efficient to use SVD: X_c = U Σ V^T, where V's columns are the principal components and Σ's diagonal entries are the square roots of the eigenvalues. SVD directly gives the PCA without forming the covariance matrix explicitly, avoiding the O(d^2) memory requirement.

The geometric interpretation

PC1 is the line (or direction) that minimises the sum of squared perpendicular distances from all data points to the line — equivalently, it maximises the variance of projections onto the line. PCA finds the best-fitting subspace, the subspace that captures as much variability as possible.

This is why PCA is a lossy compression: projecting to k < d dimensions loses the variance in the d-k remaining directions. If those directions have small eigenvalues (small variance), the loss is negligible. If they have large eigenvalues, the projection discards important variation.

Choosing k: the scree plot and explained variance

Plot the eigenvalues in decreasing order. Look for an elbow — a point where adding more components gives diminishing returns. The cumulative explained variance plot shows what fraction of total variance is captured by the top-k components. Common thresholds: keep k components that explain 90% or 95% of variance.

When PCA fails

PCA finds linear structure. If the meaningful variation in your data is nonlinear (a Swiss roll, a circle, a manifold), PCA will not find it. Kernel PCA, UMAP, and t-SNE handle nonlinear dimensionality reduction. PCA also fails when the features with highest variance are not the most predictive — variance and relevance are not the same thing. In supervised settings, use PLS (Partial Least Squares) or supervised dimensionality reduction.

PCA is sensitive to scale: features measured in different units have different variances, and PCA will be dominated by the high-variance feature. Always standardise (zero mean, unit variance) before applying PCA unless the features are naturally commensurate.

Anomaly detection with PCA

Reconstruct each data point through the k-component PCA approximation. The reconstruction error = ||x - X_pca||^2. Points with high reconstruction error lie outside the subspace captured by the principal components — they are outliers in the "unusual" dimensions. This is used for network intrusion detection, industrial sensor anomaly detection, and credit card fraud.

Try on Colab: load the MNIST dataset (784 features per image). Apply PCA and reduce to 2 dimensions. Visualise with a scatter plot, coloured by digit class — observe the clusters. Then plot cumulative explained variance vs k. Find the k needed for 90% variance. Compare reconstruction quality at k=2, k=50, k=200. Finally, use reconstruction error to detect the 1% most anomalous images — inspect them.