PCA — Principal Component Analysis

Eigenvectors, explained variance, when to use and when it fails

Imagine photographing a chair so someone can recognise it. You would not shoot it dead-on, where it collapses into a flat rectangle — you would pick the angle that shows the most at once: legs, seat, back. You are choosing the *viewpoint that keeps the most information*. Principal Component Analysis (PCA) does exactly this for data: it finds the best angle to view your features from, so that when you flatten them down to just a few numbers you keep as much of the variation as possible.

Here is where you need it. Say you have gene-expression data: 20,000 gene readings for only 500 patients. Feeding 20,000 features into a model with 500 examples is hopeless — far too many knobs, the features overlap heavily, and it is painfully slow. PCA squeezes those 20,000 down to maybe 20–50 new features that still capture over 90% of the variation — and, crucially, the new features do not overlap with each other.

The one idea: find the directions of most spread

Picture your data as a cloud of points. In some directions the cloud is stretched out; in others it is thin. PCA finds the direction of *maximum spread* — that is the first principal component. Then the next direction of most spread that sits at a right angle to the first — the second component. And so on. Each point can then be described mostly by where it sits along these few directions, instead of by all 20,000 original numbers.

Each component comes with a number, its explained variance — the share of the total spread it accounts for. Add them up and you can say "the first 30 components capture 92% of everything," which is how you decide how many to keep. (Under the hood these directions are the eigenvectors of the data's covariance matrix, and their explained-variance numbers are the eigenvalues; libraries compute them with the SVD, which is more numerically stable. You do not need the machinery to use PCA well — but that is what is happening.)

You can also run PCA in reverse: from the few components, approximately rebuild the original features. In fact PCA is, provably, the *best possible* straight-line way to compress and rebuild — no linear method loses less information for the same number of components.

The trap: PCA keeps *variance*, not *signal*.

Here is the mistake almost everyone makes: "PCA removes noise." It does not. PCA keeps the *highest-variance* directions and throws away the low-variance ones — and it has no idea which of those is signal and which is noise. Sometimes the biggest source of variation in genomics data is a *batch effect* (which day the sample was processed), pure noise that PCA will lovingly preserve. And sometimes the thing you actually care about — a rare but important pattern — has *low* variance, so PCA quietly deletes it. So never assume PCA kept what matters: always compare a PCA-reduced model against the full-feature model on the real downstream task before you trust it.

And one setup detail you cannot skip: standardise your features first. PCA chases variance, so if income is measured in dollars (variance in the billions) and another feature is a 0/1 flag, income will dominate every component for no good reason. Put everything on the same scale before running PCA, every single time.

PCA leaks too — fit it on the training fold only

PCA looks unsupervised and therefore safe, but it *learns from the data* (the components come from the covariance of the whole set). Fit PCA on all your data before splitting and the training rows already "know" the directions defined partly by the test rows — a genuine leak that inflates your score. The rule is the same as for scalers and imputers: fit PCA on the training fold, then transform validation/test with those fixed components. Wrap it in a scikit-learn Pipeline so cross-validation re-fits it inside every fold. And remember standardisation is part of this — fit the scaler on train too, not the full set.

Whitening: decorrelate and equalise

By default PCA gives you decorrelated components with *different* variances (PC1 has the most). Whitening additionally rescales every component to unit variance, so the output is fully decorrelated *and* isotropic — sometimes what a downstream model wants. But there's a catch: whitening blows the low-variance components (which are often mostly noise) up to the same scale as the high-variance ones, so it can amplify noise. Use it when the downstream method assumes equal-variance inputs; skip it when the low-variance directions are junk you'd rather keep small.

The assumptions, stated plainly

PCA rests on a few things worth naming: it's linear (it can only find straight-line directions — curved structure is invisible to it), it's variance-based (it equates "important" with "high spread," which isn't always true), and it's sensitive to outliers (a few extreme points can swing a component, since variance squares distances). If your structure is nonlinear, your signal is low-variance, or your data has heavy outliers, PCA's assumptions are working against you.

Interpretability: loadings, but not business-readable

Each component is a linear mixture of all your original features (its loadings are the weights). You *can* inspect loadings — "PC1 is mostly income and home value" — but a component like "0.4·income − 0.2·age + 0.31·tenure − …" rarely maps to something you can explain to a stakeholder. So PCA trades away the plain interpretability that feature *selection* keeps. When explanations matter (regulated lending, medicine), prefer selection over reduction.

Big or sparse data: randomized and truncated SVD

Classic PCA forms the full covariance matrix, which is expensive or impossible for very high-dimensional data (text with tens of thousands of terms). Two variants fix this: randomized PCA approximates the top components far faster, and TruncatedSVD works *directly on sparse matrices* without centering (so it doesn't destroy sparsity) — this is the standard "LSA" move for TF-IDF text. For high-dimensional or sparse inputs, reach for these rather than vanilla covariance PCA.

Visualisation is not proof

A 2D PCA scatter (PC1 vs PC2) is great for a *rough* look — spotting gross structure or obvious outliers. But it captures only the top two directions, so it is not proof of separability or cluster quality: classes that overlap in the PCA plot may separate cleanly in the full space, and apparent clusters may be artefacts. Use the plot to generate hypotheses, then validate on the real task — never conclude "the classes aren't separable" from a PCA picture.

The alternatives map

Match the tool to the goal. For visualisation of nonlinear structure, t-SNE or UMAP (they preserve local neighbourhoods far better than PCA's two axes). For nonlinear preprocessing, kernel PCA or an autoencoder (nonlinear compression). For keeping explainable original variables, feature selection instead of PCA. PCA remains the fast, stable default for *linear* compression and decorrelation — just don't force it onto jobs its assumptions don't fit.

Key points

• Choose the number of components from the cumulative explained-variance curve — but confirm it on the real task. A good starting rule is to keep enough components to cover 90–95% of the variance; plot the curve and look for the elbow where extra components stop adding much. Treat that as a starting point, not gospel — the signal you actually care about might live in a lower-variance component the rule would drop, so always check downstream performance at a few different component counts.
• The trap: forgetting to standardise before PCA. PCA maximises variance, so if your features sit on wildly different scales, the biggest one (income in dollars, say) dominates every component regardless of how useful it is, and everything else gets crushed into components you later discard. Standardise every feature first (mean 0, unit variance) — no exceptions, unless the features are already on one scale and you genuinely want the big ones to dominate.
• The diagnostic: if the PCA-reduced model does clearly worse than the full one, the signal was in a low-variance direction — or the structure is not linear. First try keeping more components. If that does not help, PCA's straight-line assumption may be the problem: the important structure could be curved, which PCA cannot capture. For that, reach for a nonlinear method — UMAP for visualising, or kernel PCA for preprocessing. A drop in performance after PCA is telling you something real about where your signal lives.
• Fit PCA inside the split, know whitening, and match the variant to the data. PCA learns from data, so fit it (and the scaler) on the training fold only — inside a Pipeline within CV — or the components leak test information. Whitening rescales all components to unit variance (decorrelated and isotropic) but amplifies the low-variance noise directions, so use it only when the downstream model wants equal-variance inputs. For high-dimensional or sparse data (TF-IDF text), use randomized PCA or TruncatedSVD (works on sparse matrices without centering) rather than covariance PCA.
• Respect PCA's assumptions and reach for the right alternative. PCA is linear, variance-based, and outlier-sensitive, and its components are linear mixtures whose loadings you can inspect but rarely explain to a stakeholder — so prefer feature selection when explainability matters. A 2D PC1-vs-PC2 plot is for rough structure and outlier spotting, not proof of separability or cluster quality (validate on the real task). For nonlinear structure use t-SNE/UMAP (visualisation) or kernel PCA / autoencoders (nonlinear compression).

Recap

• Finds directions of maximum spread (principal components), orthogonal, ranked by explained variance.
• Components = eigenvectors of covariance; computed via SVD; best linear compression that exists.
• Keeps variance, not signal: batch effects preserved, low-variance signal deleted.
• Standardise first, every time — PCA chases variance, big-scale features dominate.
• PCA leaks: fit on the training fold only, inside a Pipeline within CV.
• Whitening rescales components to unit variance but amplifies low-variance noise.
• Linear, variance-based, outlier-sensitive; 2D plot is for hypotheses, not proof of separability.

Check your understanding