Linear Algebra for ML: Eigenvalues, SVD, and Why They Are Everywhere

PCA, attention, word2vec, recommendation systems, and neural network weight initialisation all depend on the same 4 linear algebra ideas: linear transformations, eigendecomposition, SVD, and dot products as similarity. This post builds those four from scratch, in order, in the 90 minutes it would have taken your university course if it started with applications.

Linear algebra is the operating system of machine learning. Matrix multiplication is the operation every neural network layer performs. Eigendecomposition is what PCA computes. SVD is what collaborative filtering and LSA are built on. Dot products are the similarity measure in attention and in nearest-neighbour search. You cannot deeply understand any of these without the linear algebra underneath.

Vectors: what they actually are

A vector is a list of numbers that represents a point or a direction in some space. In ML, a row in your dataset is a vector in feature space — a point in ℝᵈ where d is the number of features. The dot product u · v = Σᵢ uᵢvᵢ = ||u|| ||v|| cos(θ) where θ is the angle between u and v. This geometric interpretation is the reason dot products measure similarity: if two vectors point in the same direction, cos(θ) = 1 (maximum similarity); if orthogonal, cos(θ) = 0 (no similarity); if opposite, cos(θ) = -1. Every attention mechanism and every embedding similarity search uses this fact.

Matrices as linear transformations

A matrix A ∈ ℝᵐˣⁿ is a function that maps vectors from ℝⁿ to ℝᵐ. The transformation is linear: A(u + v) = Au + Av. What does this mean geometrically? Every matrix transformation does some combination of rotation, scaling, shearing, and projection. No curving allowed — that is what "linear" means. A neural network layer with no activation function is a linear transformation. Stacking multiple linear layers gives another linear layer — which is why we need nonlinear activations to make deep networks more expressive than a single layer.

Eigenvalues and eigenvectors

For a square matrix A, vector v is an eigenvector with eigenvalue λ if Av = λv. The matrix transforms v by pure scaling (by λ) rather than rotating it. Geometrically: eigenvectors are the "axes" of the transformation — the special directions that don't get rotated, only stretched or flipped. Positive λ: stretching. Negative λ: flipping. |λ| > 1: expanding. |λ| < 1: contracting. |λ| = 0: collapsing to zero (the matrix is rank-deficient). For a symmetric matrix A = Aᵀ (like a covariance matrix), all eigenvalues are real and eigenvectors are orthogonal. This is the spectral theorem. Covariance matrices are always symmetric — which is why PCA's eigenvectors (principal components) are orthogonal.

PCA as eigendecomposition of the covariance matrix

Given data matrix X ∈ ℝⁿˣᵈ (n samples, d features), the covariance matrix C = (1/n) XᵀX ∈ ℝᵈˣᵈ. PCA finds the directions of maximum variance, which are the eigenvectors of C with the largest eigenvalues. Eigendecomposition: C = VΛVᵀ where V is the matrix of eigenvectors (columns) and Λ = diag(λ₁, ..., λᵈ) with λ₁ ≥ λ₂ ≥ ... ≥ λᵈ. Project data onto the top-k eigenvectors: X_reduced = X V_k. Explained variance ratio of component i: λᵢ / Σⱼ λⱼ. Plot the cumulative explained variance (scree plot) to choose k.

Singular Value Decomposition (SVD)

SVD generalises eigendecomposition to rectangular matrices: A = UΣVᵀ, where U ∈ ℝᵐˣᵐ (left singular vectors), Σ ∈ ℝᵐˣⁿ (diagonal, singular values σ₁ ≥ σ₂ ≥ ... ≥ 0), Vᵀ ∈ ℝⁿˣⁿ (right singular vectors). The singular values measure how much of the "signal" lives in each dimension. Low-rank approximation: A ≈ U_k Σ_k V_k ᵀ (keep only the top k singular vectors/values) is the best rank-k approximation in the Frobenius norm (Eckart-Young theorem). Where SVD appears in ML: PCA — SVD of the data matrix X gives the same result as eigendecomposition of XᵀX (faster and numerically more stable). Collaborative filtering — the user-item rating matrix R ≈ UΣVᵀ; the product gives reconstructed ratings and reveals latent factors. Latent Semantic Analysis (LSA) — SVD of the term-document matrix reveals latent topic structure. Word2vec training is closely related to SVD of the PMI matrix (Levy & Goldberg, 2014). Neural network weight matrices have their largest singular values regulated by spectral normalisation in GANs.

The rank of a matrix

Rank = number of linearly independent columns (= number of linearly independent rows = number of non-zero singular values). A matrix is full-rank if rank = min(m, n). A rank-deficient matrix cannot be inverted — this is the multicollinearity problem in regression. When your feature matrix X has rank less than d (more features than independent information), OLS is undefined and you need regularisation (ridge, lasso) or dimensionality reduction.

Matrix norms

Frobenius norm: ||A||_F = √(Σᵢⱼ Aᵢⱼ²) = √(Σᵢ σᵢ²). The sum of squared entries. Used in L2 regularisation on weight matrices. Spectral norm: ||A||₂ = σ₁ (largest singular value). Controls how much a layer can amplify a signal. Used in spectral normalisation for GANs. Nuclear norm: ||A||_* = Σᵢ σᵢ. Promotes low-rank solutions when used as a regulariser (matrix completion).

Interview questions on this topic

"What does it mean for two vectors to be orthogonal? Why does this matter in PCA?" — Their dot product is zero, cos(θ) = 0, they share no similarity. PCA components are orthogonal because the covariance matrix is symmetric (spectral theorem) — each component captures variance that no other component captures.

"You have a user-item rating matrix with 10M users and 1M items and 0.01% fill rate. How would you use SVD?" — Truncated SVD on the sparse matrix (using scipy.sparse.linalg.svds) to get rank-k factorization. The left singular vectors are user embeddings, right singular vectors are item embeddings. Predict missing ratings as the dot product. This is exactly matrix factorisation collaborative filtering.

"What is the relationship between PCA and SVD?" — SVD of the centred data matrix X = UΣVᵀ gives principal components as columns of V and principal component scores as columns of UΣ. Eigendecomposition of XᵀX = V(Σ²)Vᵀ gives the same V. SVD is preferred numerically.

"Why does vanishing gradient happen in terms of linear algebra? How does it connect to singular values?" — During backprop, gradients are multiplied by Jacobians. If the Jacobians have small singular values (nearly rank-deficient), gradient magnitudes shrink exponentially with depth. Large singular values cause exploding gradients. Proper initialisation (He, Xavier) sets the singular value distribution of the initial weight matrices so gradients stay in a stable range.

Try on Colab: take the MovieLens 100K dataset. Build the user-item rating matrix (sparse). Apply truncated SVD with k=10, 50, 100 components. Measure reconstruction error on observed ratings. Plot the singular value spectrum. Compare the top singular vector as a "latent genre" — which movies load most heavily on component 1?