Vectors & Matrices
Dot product, matrix operations, rank, norms
You have 10,000 training images, each 64×64 pixels × 3 channels = 12,288 numbers per image. These 10,000 images form a matrix X ∈ ℝ^{10000 × 12288}. Every ML operation on this dataset — normalizing features, computing covariances, running regression, doing PCA, forward passes through a neural network — is a matrix operation. If you do not know what a matrix product computes geometrically, you are manipulating ML pipelines you cannot understand or debug.
Vectors carry direction and magnitude. The dot product a·b = ‖a‖ ‖b‖ cos(θ) measures directional alignment — it is large and positive when two vectors point in similar directions, and zero when they are perpendicular. Orthogonal vectors have dot product zero. This is why cosine similarity works for semantic search: aligned embeddings have high dot products, reflecting similar meaning.
Matrix multiplication AB = C means C[i,j] = row i of A dotted with column j of B. It is not commutative: AB ≠ BA in general. Dimensions must match: A (m×k) times B (k×n) gives C (m×n). A matrix is a linear transformation — it stretches, rotates, and projects vectors. The matrix inverse A⁻¹ satisfies A⁻¹A = I. For linear regression, the normal equations give θ = (X^T X)⁻¹ X^T y — the closed-form solution via the Gram matrix X^T X.
Column space of A: all vectors reachable as Ax for some x. Rank: the dimension of the column space — the number of linearly independent directions A can produce. Rank deficiency means X^T X is singular, meaning the normal equations have no unique solution. You need regularization to fix this.
A norm ‖v‖ measures a vector's length; different norms make different tradeoffs. The L2 (Euclidean) norm ‖v‖₂ = √(∑ᵢ vᵢ²) is the one used in the dot product formula above — for v = [3, 4], ‖v‖₂ = √(9+16) = 5. The L1 norm ‖v‖₁ = ∑ᵢ|vᵢ| sums absolute values (for the same v, 3+4 = 7); minimizing it (Lasso) drives some coefficients to exactly zero, because its penalty grows at a constant rate per coordinate instead of shrinking smoothly toward zero. The L∞ norm ‖v‖∞ = maxᵢ|vᵢ| just takes the largest coordinate. Which norm a loss function penalizes changes what solutions look like: L2 regularization (Ridge, below) shrinks every coefficient a little; L1 regularization (Lasso) zeroes some out entirely.
NOT this. Linear algebra is not just matrix arithmetic for solving linear systems. Every gradient descent step in a neural network is a matrix-vector multiplication. Every embedding lookup is a dot product. PCA is eigenvector decomposition of the covariance matrix. Attention is softmax(QK^T / √d)V — two matrix products (QK^T, then the softmax output times V). The Gram matrix X^T X appears in every regularized linear model. Linear algebra is the operational language of every ML computation, and understanding it geometrically — as transformations of space — is what lets you reason about what information your model is processing.
Key points
- When a matrix operation fails or gives unexpected results, check dimensions first. Almost all linear algebra bugs are shape mismatches. Write out the expected shape of every tensor before the operation and verify them in code. In numpy, `.shape` before every new operation is not paranoia — it is the cheapest debugging step you have.
- Trap: computing the matrix inverse to solve Ax = b. The inverse A⁻¹ is numerically unstable for ill-conditioned matrices and costs O(n³) to compute. Use scipy.linalg.solve(A, b) instead — it uses LU decomposition, is faster, and is more numerically stable. The only reason to compute A⁻¹ explicitly is if you need to multiply by the same inverse many times.
- Diagnostic: if X^T X is near-singular (condition number — the ratio of its largest to smallest singular value — greater than 1e10), you have multicollinear features or rank-deficient data. Add L2 regularization to form X^T X + λI, which is always invertible for λ > 0. This is exactly what Ridge regression does — but for any fixed λ > 0, Ridge's solution (X^T X + λI)⁻¹X^T y is a biased, shrunk estimator, not the minimum-norm solution to the original normal equations. The true minimum-norm solution (via the Moore–Penrose pseudoinverse) is only the λ→0⁺ limit of the ridge family — it is not what Ridge computes at any regularization strength actually used in practice.
Every ML forward pass is matrix multiplication and nonlinearities. Rank tells you where information is irreversibly lost. Norms tell you what geometry an algorithm assumes. Both predict failure modes before you run a single experiment.
Recap
- Every ML op is a matrix op: normalise, covariance, regression, PCA, forward pass — all linear algebra.
- Dot product $a\cdot b=\|a\|\|b\|\cos\theta$ measures alignment; 0 = orthogonal. Basis of cosine similarity.
- Matrix product = row·column; not commutative (AB ≠ BA); shapes must match (m×k)(k×n)→(m×n).
- Rank = number of independent directions. Rank-deficient X → XᵀX singular → normal equations have no unique solution.
- Normal equations: $\theta=(X^TX)^{-1}X^Ty$ — the Gram matrix XᵀX appears in every linear model.
- Debug shapes first — most linear algebra bugs are shape mismatches. Don't invert to solve Ax=b; use `solve`.
- Near-singular XᵀX (κ, the ratio of largest to smallest singular value, > 1e10) = multicollinearity; Ridge adds λI to make it invertible — but Ridge is a biased, shrunk estimator, not the min-norm fix (the true min-norm solution is only the λ→0⁺ limit).
Check your understanding
Q1. A system Ax=b where A is 3×5 (3 equations, 5 unknowns). What can you say about the solution set? When does a solution exist?
- A) A is 3×5 with rank r ≤ min(3,5) = 3. If rank(A) = 3 (full row rank): Ax=b is consistent for every b, since col(A) spans ℝ³. The solution is not unique — there are 5−3 = 2 free variables, so the solution set is a 2-dimensional affine subspace of ℝ⁵. If rank(A) < 3, some b outside col(A) has no solution.
- B) A is 3×5 so the system has more equations than unknowns — it is overdetermined and generically has no exact solution. We use least squares: x̂ = (AᵀA)⁻¹Aᵀb. AᵀA is 5×5 and full rank when A has rank 5, so the system is uniquely solvable via the normal equations, with residual r = b − Ax̂ orthogonal to every column of A by construction.
- C) A is 3×5 with 3 equations and 5 unknowns, giving 5−3=2 degrees of freedom. A solution always exists because the system is underdetermined — there are always more unknowns than equations, so b is always reachable regardless of rank. The unique minimum-norm solution is x̂ = Aᵀ(AAᵀ)⁻¹b, found by inverting the 3×3 Gram matrix AAᵀ directly.
- D) A is 3×5, so the column space of A has dimension at most 5, since a matrix's column space dimension equals its number of columns. Ax=b has a unique solution whenever rank(A)=3 and b is in the column space, and infinitely many solutions when rank(A)<3. There are always exactly 5−3=2 free variables in either case, independent of b.
Q2. You compute the dot product of two vectors: u·v = ‖u‖‖v‖cos(θ) = 0. What does this mean geometrically, and what does it mean in ML for feature representations?
- A) u·v = 0 means either u = 0 or v = 0 — at least one vector is the zero vector, since the zero vector has undefined direction and magnitude 0. In ML, if a feature representation u = 0, that data point has no learned embedding and receives uniform attention weights from every query, making it effectively invisible to the model.
- B) u·v = 0 means the vectors have equal magnitude (‖u‖ = ‖v‖). Geometrically, equal-length vectors that differ only in direction have zero dot product when aligned symmetrically around the origin. In ML, this means the two representations encode the same amount of information but in completely different directions.
- C) u·v = 0 means u and v are linearly dependent — one is a scalar multiple of the other, i.e. v = ku for some real k. Geometrically, they point in exactly the same or exactly opposite directions. In ML, linearly dependent feature vectors indicate that two data points have proportional feature activations — they lie on the same ray through the origin, differing only by a scale factor.
- D) u·v = 0 means θ = 90° — the vectors are orthogonal, geometrically perpendicular in ℝⁿ. In ML, if u and v are feature representations of two points, this means the features share no common activation pattern. For embeddings, u·v = 0 means the two entities have no learned similarity; after normalising, cos(θ)=u·v/(‖u‖‖v‖) is the cosine similarity, 0 at orthogonality.
Q3. The matrix A = [[2, 1], [4, 2]] is singular. Which TWO of the following statements about it are true?
- A) det(A) = 2×2 − 1×4 = 0, so A is singular; equivalently, row 2 = 2×row 1, so the rows are linearly dependent. The null space of A is spanned by v = [1, −2]ᵀ (since Av = 0) — this is exactly the direction that collapses to zero, so any two inputs differing by a multiple of v produce the same output Ax.
- B) det(A) = 2×2 − 1×4 = 0 → A is singular, with rank 1 — the column space is a one-dimensional line through the origin in ℝ². For Ax=b: if b lies on that line there are infinitely many solutions (a 1D affine subspace); if b does not lie on that line, no solution exists at all, since b is unreachable.
- C) Row 2 = 2 × row 1, so the rows are linearly dependent and det(A) = 0. For Ax=b: the system always has infinitely many solutions because the null space is non-trivial — the extra degree of freedom means we can always shift any particular solution by a null space vector, so the system is never inconsistent for any b.
- D) det(A) = 4 − 4 = 0 and trace(A) = 2+2 = 4. Since A is singular with equal diagonal entries, both eigenvalues equal 2 — the characteristic polynomial factors as (λ−2)² = 0. For Ax=b: the matrix is rank-deficient, so we need the pseudoinverse A⁺ = VΣ⁺Uᵀ to get the minimum-norm least-squares solution.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →