PCA: The Intuition No One Teaches

Every ML course covers PCA. Very few explain it in a way that builds genuine intuition. What does "variance explained" actually mean? Why does the first principal component always capture the most variance? And when should you NOT use PCA?

PCA is fundamentally about finding a new coordinate system where the axes align with the directions of maximum variance in your data.

The geometric view:

Imagine a cloud of points in 3D space. The cloud is elongated in one direction (high variance along axis 1), somewhat spread in a second direction (moderate variance along axis 2), and nearly flat in the third (low variance). PCA finds these three directions — PC1, PC2, PC3 — and orders them by variance explained.

Each principal component is a linear combination of your original features. PC1 = a₁×feature₁ + a₂×feature₂ + ... + aₙ×featureₙ. The coefficients are the eigenvectors of the covariance matrix. The variance explained is the corresponding eigenvalue.

The covariance matrix view:

Compute Σ = (1/n) × XᵀX (on mean-centered, standardised data). SVD: Σ = UΛUᵀ. The columns of U are the principal components. The diagonal of Λ contains the variances. This is why you always standardise before PCA — features on different scales will have their variance dominated by the highest-magnitude features, and PCA will find them first regardless of information content.

When PCA hurts:

When your task is classification. PCA maximises variance, not class separability. A feature with high variance might be pure noise. LDA (Linear Discriminant Analysis) maximises class separability instead — that's what you want for supervised dimensionality reduction.

When interpretability matters. PC1 = 0.4×age - 0.3×income + 0.7×account_age ... is not something you can explain to a product team or a regulator. If your features need to remain interpretable, use feature selection (RFE, SHAP-based) instead of PCA.

When your data is sparse. PCA is dense by construction. For text (TF-IDF matrices), image features, or any high-dimensional sparse data, truncated SVD (LSA) or sparse PCA will work better.

The scree plot and the elbow rule:

Plot eigenvalues in descending order. The "elbow" — where the curve flattens — is typically a good cutoff. But rules of thumb vary by domain: "keep 90% of variance" is common but arbitrary. Better: keep enough components that cross-validation performance on your downstream task doesn't degrade significantly when you add more.