Feature Selection & Dimensionality

Filter/wrapper/embedded, mutual information, RFE, SHAP

It feels obvious that more features should mean better predictions — more information about each house, each patient, each customer. For a while that is true. But there is a point where piling on features starts making the model *worse*, and the reason has a wonderful name, coined by the mathematician Richard Bellman in the 1950s: the curse of dimensionality.

Here is the curse in one picture. Imagine 100 data points spread along a single line (one feature) from 0 to 1 — they sit packed close together, a few hundredths apart. Now give each point a second feature: the same 100 points scatter across a square, and the gaps between them widen. Add a third feature and they float in a cube, mostly empty space around each one. Keep going, and in high dimensions the points drift so far apart that every point is roughly the same enormous distance from every other — there is simply not enough data to fill the space. Patterns that were obvious in a few dimensions dissolve into the emptiness.

So extra features are not free. Irrelevant ones add noise for the model to trip over. Redundant ones (two features saying the same thing) add nothing but confusion. And every extra dimension thins out your data, which makes overfitting easier and real structure harder to find. The cure is to be deliberate about which features you keep — that is feature selection.

Three families of ways to choose

There are three broad strategies, trading speed for smartness.

Filter methods are the quick screen. They score each feature on its own — how strongly does it relate to the target, using a measure like correlation or mutual information — and keep the top ones. Fast and simple, but blind: they judge each feature in isolation, so they can throw away two features that are useless alone yet powerful together.

Wrapper methods are the brute-force search. They actually train the model on different subsets of features and keep whichever subset scores best. A common version, recursive feature elimination, trains the model, drops the weakest feature, and repeats. Accurate, because the model itself is the judge — but slow, since it means training many times, which becomes impractical with thousands of features.

Embedded methods fold selection into training itself. Lasso (L1 regularisation) is the classic: as the model trains, its penalty drives useless features' weights all the way to zero, selecting and fitting at once. Tree-based importance — or better, SHAP values — does something similar: a single trained model ranks the features it actually leaned on, interactions included. For everyday tabular work this is the most reliable recipe: train one gradient-boosted model on everything, rank the features by importance, and keep the top ones.

Selection versus reduction — a real distinction

There is a cousin of feature selection worth separating out: dimensionality reduction, of which PCA is the famous example. The difference matters. Selection *keeps a subset of your original features* — afterward you can still say "we used income, age, and debt." Reduction *invents new features* by blending the old ones into a handful of combined directions that capture most of the variation. Those directions are compact but no longer mean anything you can explain to a person ("0.4 × income minus 0.2 × age…"). So choose selection when you need to keep things explainable, and reduction when you only care about squeezing the information into fewer numbers.

The one trap that fakes good results

Whatever method you use, there is a mistake that quietly inflates your numbers: choosing features using the whole dataset, before you split off a test set. If you pick features by how they relate to the label across *all* your data, you have let the test set's answers leak into the choice — the chosen features look more predictive than they really are, and your reported accuracy is a mirage. The fix is to do feature selection *inside* your cross-validation: choose features using only the training portion of each fold, and judge on the held-out portion. Select on the training data only. Always.

SHAP is powerful but not gospel

SHAP is the best-known importance tool, and it's genuinely good — it fairly attributes a prediction across features with solid theory behind it. But two caveats keep you honest. First, under correlated features SHAP can *split* or *shuffle* credit between the correlated group in ways that mislead — a truly important feature can look weak because its correlated twin absorbed the attribution. Second, SHAP explains what the *model* used, not what *causes* the outcome — high SHAP importance is not evidence of causality. Treat SHAP as "what is this model leaning on," never as "what drives the world."

Permutation importance, and its blind spot

A model-agnostic alternative worth naming: permutation importance shuffles one feature's values and measures how much accuracy drops — a big drop means the model really relied on it. It works on any fitted model and needs no retraining. Its blind spot is the same correlation trap that haunts forests: with two correlated features, shuffling one barely hurts because its twin still carries the signal, so both look unimportant. So read permutation importance with the correlation caveat, and consider dropping correlated groups together to test them jointly.

Mutual information, read carefully

Correlation only catches *linear* relationships; mutual information catches *any* dependence, linear or not, which is why it's a better filter score. But mind its limits: as usually applied it's univariate (scores each feature against the target alone, missing features that only matter in combination), and its estimate is sensitive to binning, the estimator, and sample size — noisy MI values with little data can rank features almost at random. Useful as a fast screen, not a final verdict.

RFE in practice: use RFECV

Recursive feature elimination is accurate but has practical costs: it's expensive (retrains repeatedly), estimator-dependent (the ranking changes with the model you wrap), and unstable under correlated features. And plain RFE makes you guess *how many* features to keep. The fix is RFECV — wrap RFE in cross-validation so it *selects the feature count* by held-out performance instead of you picking it by hand.

PCA's fine print

If you do reach for PCA, know its assumptions. It's unsupervised — it keeps the directions of largest *variance*, which are not necessarily the directions that predict your target (a high-variance feature can be pure noise). It's scale-sensitive, so you must standardise first or the largest-unit feature dominates the components. And the components are linear blends of everything, so they're hard to explain. PCA reduces dimensions and decorrelates, but it can throw away exactly the low-variance signal that mattered.

When dimensions explode: special regimes

The right strategy shifts sharply with the data type. For text / sparse one-hot features (tens of thousands of mostly-zero columns), L1/Lasso-style selection and sparse-aware methods fit naturally. For genomics (p ≫ n, thousands of genes, few samples), univariate screening plus stability selection is common. For embeddings (dense learned vectors), individual dimensions are meaningless, so you reduce or regularise rather than select individual columns. Don't apply a tabular feature-selection recipe blindly to text, genomic, or embedding data.

Is the selection even stable?

One last discipline: a feature set chosen from a single run can be a fluke of that particular sample. Stability selection checks this — rerun the selection on many bootstrap resamples (or CV folds) and keep the features that get chosen *consistently*. If a feature appears in 90% of runs, trust it; if it flickers in and out across runs, it's likely noise dressed up as signal. Stable selections generalise; one-run selections often don't.

Key points

• More features is not always better — past a point they add noise and thin out your data (the curse of dimensionality). Irrelevant features give the model more ways to trip; redundant ones add confusion; and every extra dimension spreads your data points further apart, which makes overfitting easier and real patterns harder to find. So be deliberate about which features you keep. A solid default recipe: train one gradient-boosted model on everything, rank the features by importance (SHAP is a strong choice, though read with care under correlated features), and keep the top ones.
• Know the three ways to choose, and their tradeoff of speed versus smartness. Filter methods score each feature on its own — fast, but blind to features that only matter in combination. Wrapper methods train the model on different subsets and keep the best — accurate, but slow, and impractical with thousands of features. Embedded methods select while training: Lasso zeros out useless weights as it fits, and tree or SHAP importance ranks what a trained model actually used. For everyday tabular work, embedded methods give the best balance of the three.
• The trap that fakes good results: choosing features on the whole dataset before splitting off a test set. If you pick features by how they relate to the label across all your data, the test set's answers have leaked into the choice, and your reported accuracy is a mirage. Do feature selection inside cross-validation: choose features using only the training portion of each fold, then judge on the held-out portion. And keep selection (which preserves your original, explainable features) separate from dimensionality reduction like PCA (which blends them into compact but unexplainable new ones).
• Read every importance method with the correlation caveat, and don't confuse importance with cause. SHAP fairly attributes predictions but splits credit between correlated features and shows what the *model* used, not what *causes* the outcome — high SHAP ≠ causal. Permutation importance is model-agnostic but hits the same correlation trap (shuffling one of a correlated pair barely hurts). Mutual information catches non-linear dependence but is univariate and binning/sample-sensitive. RFE is accurate but expensive, estimator-dependent, and unstable — use RFECV to pick the feature count. And PCA keeps high-variance directions, which aren't necessarily predictive, needs standardising, and yields unexplainable components.
• Confirm your selection is stable, and adapt to the data regime. A feature set from one run can be a fluke — stability selection reruns the choice across bootstraps/folds and keeps features chosen consistently (appears in 90% of runs → trust it; flickers → noise). And the recipe shifts with the data: L1/sparse methods for text and one-hot, univariate screening plus stability selection for genomics (p ≫ n), and reduce/regularise rather than select individual columns for dense embeddings — a tabular recipe doesn't transfer blindly.

Recap

• More features isn't always better — past a point they add noise and thin your data out (curse of dimensionality).
• Three ways to choose, speed vs smartness: filter (fast, blind), wrapper (accurate, slow), embedded (best balance).
• Embedded = Lasso, tree or SHAP importance — usually the sweet spot.
• Selection ≠ reduction: selection keeps original explainable features; PCA invents new ones.
• Always select inside cross-validation — choose features on the whole dataset and test labels leak in (mirage results).
• Read importances with the correlation caveat, and importance ≠ cause.

Check your understanding