Eigenvalues & Eigenvectors

Geometric intuition, spectral theorem, power iteration

You have customer behavioral data with 50 features. Three of them — features 3, 7, and 22 — are all variations of "how much did the user spend." They carry similar information while consuming three times the parameter weight in regularization and three times the compute in every matrix operation. If you could find the single direction that captures the spending signal, you would reduce noise, compute, and overfitting simultaneously. That direction is a principal component — an eigenvector.

A square matrix A has eigenvalue λ and eigenvector v if Av = λv. The matrix A acts on v by pure scaling — the direction does not change, only the magnitude. For a symmetric positive semidefinite matrix like a covariance matrix, all eigenvalues are ≥ 0 and all eigenvectors are orthogonal. This is what makes PCA geometrically clean: the principal components are mutually perpendicular directions.

The covariance matrix C = X^T X / (n-1). Its eigenvectors are the principal directions of variance in the data. Its eigenvalues tell you how much variance lies along each direction. The first eigenvector, with the largest eigenvalue, is the direction of maximum variance — this is PC1. The second eigenvector, orthogonal to PC1, is PC2. The eigenvalue ratio λ₁ / Σλᵢ tells you the fraction of total variance captured by PC1.

Spectral theorem: any real symmetric matrix A decomposes as A = Q Λ Q^T where Q is orthogonal (columns are eigenvectors) and Λ is diagonal (eigenvalues on diagonal). This decomposition separates the rotating part (Q) from the scaling part (Λ). For PCA, Q gives you the rotation into principal component space, and Λ tells you how much each direction matters. This "orthogonal matrix's columns are eigenvectors" fact is special to symmetric A, though — it is not a general property of orthogonal matrices. A generic orthogonal matrix (a rotation, say) is not diagonalized by its own columns, and can have no real eigenvectors at all: the rotation R = [[0,−1],[1,0]] (a 90° turn) solves det(R−λI) = λ²+1 = 0, giving λ = ±i — no real eigenvalue, because no real vector keeps its direction under a genuine rotation; every real vector just gets turned 90°. The general 2D rotation R(θ) = [[cos θ,−sin θ],[sin θ,cos θ]] has eigenvalues λ = cos θ ± i·sin θ = e^{±iθ}, real only at θ=0 (λ=1, R=I) or θ=180° (λ=−1, R=−I). Symmetric matrices are guaranteed real eigenvalues and orthogonal eigenvectors; non-symmetric matrices are not.

Multiplying Q Λ Q^T out column by column turns the matrix form into a sum of rank-1 matrices: A = Σᵢ λᵢ vᵢ vᵢ^T, where each vᵢvᵢ^T is a rank-1 projector — project onto the line spanned by vᵢ, then scale by λᵢ. Worked example: A = [[4,1],[1,4]] has eigenvalues λ₁=5 with eigenvector v₁=[1,1]/√2, and λ₂=3 with eigenvector v₂=[1,−1]/√2. Check the sum directly: 5·v₁v₁^T = 5·[[0.5,0.5],[0.5,0.5]] = [[2.5,2.5],[2.5,2.5]], and 3·v₂v₂^T = 3·[[0.5,−0.5],[−0.5,0.5]] = [[1.5,−1.5],[−1.5,1.5]]; adding them gives [[4,1],[1,4]] = A. So A is nothing more than "project onto v₁ and stretch by 5" plus "project onto v₂ and stretch by 3" — the sum-of-rank-1-projectors view and the QΛQ^T matrix view are the same object, just written two different ways.

NOT this. Eigenvalues are not a mathematical abstraction with limited practical use. The eigenvalues of a neural network's Hessian determine optimization dynamics: a flat loss landscape has many near-zero eigenvalues sitting alongside a few large ones, so the condition number κ = λ_max/λ_min is huge. Vanilla SGD uses one global step size, capped by the steepest direction (λ_max) so it doesn't diverge there — which forces that same tiny step size onto the near-zero-eigenvalue flat directions too, so progress along those directions crawls. Adam instead rescales each parameter's step by a running estimate of that parameter's own gradient magnitude, which approximates dividing by the local curvature per direction — so in the same update it can take a large step along a flat (near-zero-eigenvalue) direction and a small step along a steep (large-eigenvalue) direction. That per-direction rescaling — not the mere presence of near-zero eigenvalues — is why Adam with good initialization typically converges faster than vanilla SGD on the same ill-conditioned, overparameterized loss surface. The eigenvalues of the attention matrix determine how information diffuses through a transformer layer. The eigenvalue spectrum of X^T X tells you the effective dimensionality of your data before you fit any model. Eigenvalues are the fingerprint of every matrix that matters in ML.

Key points

• When features are correlated, eigendecompose the covariance matrix before modeling. The eigenvalue spectrum tells you how many truly independent directions of variation exist in your data. If 5 of 50 eigenvalues contain 90% of the variance, you can reduce to 5 components with negligible information loss — the remaining 45 components are dominated by noise and correlations, not signal.
• Trap: forgetting to center data before computing the covariance matrix. If the mean is nonzero, the first eigenvector points toward the mean rather than toward the direction of maximum variance. Always subtract the column means from X before computing X^T X / (n-1). sklearn's PCA does this by default. If you compute the covariance matrix manually, centering is mandatory.
• Diagnostic: plot the explained variance ratio (eigenvalue_i / sum of all eigenvalues) as a cumulative curve. The elbow in the curve tells you how many components to keep. Where the curve flattens, adding more components yields diminishing returns — each additional component explains only noise. A sharp elbow at k components means the data lies approximately on a k-dimensional subspace.
• Power iteration finds the dominant eigenvector without ever forming the full eigendecomposition: starting from any v₀, repeatedly apply v ← Av/‖Av‖. Expand v₀ in the eigenbasis, v₀ = Σᵢ cᵢvᵢ; then Aᵏv₀ = Σᵢ cᵢλᵢᵏvᵢ = λ₁ᵏ(c₁v₁ + Σᵢ>1 cᵢ(λᵢ/λ₁)ᵏvᵢ). Since |λᵢ/λ₁| < 1 for every i>1, those terms shrink geometrically with k and only the c₁v₁ term survives — the ‖Av‖ normalisation at each step just prevents overflow/underflow, it doesn't change which direction wins. The convergence rate is |λ₂/λ₁|ᵏ: a large spectral gap (λ₁ ≫ λ₂) converges in a handful of iterations, a near-tie (λ₁≈λ₂) converges slowly. This is the algorithm behind PageRank: the dominant eigenvector of the web's link-transition matrix is the PageRank vector, found by repeated multiplication rather than by eigendecomposing a matrix with billions of rows.

Recap

• Av = λv: the matrix scales v without rotating it. Eigenvectors are the invariant directions.
• Covariance C = XᵀX/(n−1): its eigenvectors are principal directions, eigenvalues the variance along each.
• PC1 = eigenvector with largest eigenvalue = direction of max variance; λ₁/Σλᵢ = fraction of variance captured.
• Symmetric PSD (covariance) ⟹ eigenvalues ≥ 0, eigenvectors orthogonal — this is what makes PCA clean.
• Spectral theorem: A = QΛQᵀ separates rotation (Q) from scaling (Λ).
• Center data first — otherwise the first eigenvector points at the mean, not the max-variance direction.
• Scree/explained-variance elbow tells you how many components to keep; the flat tail is noise, not signal.

Check your understanding