Feature Selection
Reduce dimensionality to fight overfitting, cut training cost, and build models that generalize.
A data warehouse dumps 500 features into your fraud model — transaction details, user behaviour, device fingerprints, merchant info, and hundreds of derived aggregates. Training on all 500 takes four hours, inference crawls at 900ms per request, and the serving box runs out of memory. You need to get under 50 features. But *which* 50?
There are four families of ways to choose, trading speed for smartness — and one trap that snares all of them.
Filter methods — score each feature alone
The fastest approach ranks each feature on its own — how strongly does it relate to fraud, by correlation or mutual information — and keeps the top ones. Cheap and parallel. The catch is baked in: it judges features *one at a time*, so it will happily throw away two features that are useless alone but powerful *together*. Interactions are invisible to it.
Wrapper methods — let the model judge
These actually train the model on different feature subsets and keep whichever scores best. Recursive feature elimination trains on all 500, drops the single weakest, retrains, and repeats down to 50 — around 450 model fits, but every decision is *model-aware*, so it can tell when two features are redundant given the others.
Embedded methods — select while training
L1 (Lasso) regularisation drives useless features' weights to exactly zero *during* fitting, so selection and training happen in one run — no separate step. The reason is geometric: L1's penalty region is a diamond whose corners sit exactly on the axes, so the optimum often lands on a corner — a weight at exactly zero. (Ridge/L2's penalty region is a smooth sphere with no corners, so the optimum rarely lands on an axis; weights shrink toward zero but almost never reach it, so it does not select.)
Permutation importance — the honest referee
Train any model, then *shuffle* one feature's values and see how much performance drops on held-out data. A big drop means the feature was pulling its weight; no drop means it was redundant or noise. This is the method to trust, because it measures real, out-of-sample usefulness — and it catches a nasty trap that tree-based importance falls into. That trap: a tree's built-in importance is biased toward *high-cardinality* columns. Feed it a `customer_id` and it will "split" on individual IDs to memorise the training set, scoring the ID as hugely important — while on new customers it is worth exactly nothing. Permutation importance on validation data exposes this instantly (shuffling the ID changes nothing), where the tree's own numbers are fooled.
The trap that snares all four: correlation is not importance
It is tempting to say "these two features are 95% correlated, drop one." Resist it. Two correlated features can still both help — keeping both can make the model steadier when one of them drifts at serving time. Correlation describes the *inputs*; it does not tell you the *predictive contribution*. So decide what to keep by measuring importance directly — permutation importance on validation data — not by eyeballing a correlation matrix.
A separate problem for linear models: multicollinearity
Keeping correlated features helps *predictive* stability, but for a *linear* model specifically, near-duplicates (`height_cm` and `height_inches`, correlated 0.97) create a different failure. The design matrix is nearly singular, so infinitely many weight combinations — a large positive weight on one feature, a canceling negative weight on the other — fit the training data almost equally well. Predictions stay fine; the *individual coefficients* become wildly unstable and impossible to interpret. This is multicollinearity, measured by VIF (variance inflation factor: `VIF = 1 / (1 - R²)`, where R² comes from regressing that feature on the rest) — a high VIF flags exactly this instability, a separate concern from the importance-vs-correlation trap above.
Selection is preprocessing — do it inside the CV folds
The trap that quietly inflates every method above: feature selection uses the labels, so it leaks if done on the full data before splitting. Rank features by correlation/MI/importance across the whole dataset, then cross-validate, and the "held-out" folds already helped choose the features — your score is optimistic. Selection (filter, wrapper, embedded, RFE, target-based) must run inside each CV fold on the training portion only, exactly like a scaler or encoder. Wrap it in a Pipeline so it can't be skipped.
Is the selection stable?
A feature set from one run can be a fluke of that sample. Stability selection reruns the choice across bootstraps or folds and keeps only the features chosen *consistently* — a feature that appears in 90% of runs is real signal, one that flickers in and out is noise dressed as signal. This matters most with correlated features, where which one gets picked can flip run to run (Lasso is notorious for this). Stable selections generalise; one-run selections often don't.
Caveats on the tools you'll reach for
Mutual information catches non-linear dependence (unlike correlation) but as usually applied is univariate (misses features that only matter in combination) and is noisy and sample-size/binning sensitive. SHAP attributes predictions well but splits credit among correlated features (a truly important feature can look weak because its twin absorbed the attribution) and shows what the *model* used, not causation — high SHAP ≠ causal. Embedded tree selection isn't just "use importance": the tree's own knobs (`min_child_weight`, `gamma`, `max_depth`, `colsample_bytree`, feature subsampling) *are* a form of regularised selection, pruning weak features during training. Read all of them with the correlation caveat.
Selection versus reduction, and the cost you forget
Keep feature selection (keeps a subset of your *original, explainable* features) separate from dimensionality reduction like PCA (invents new latent components — compact but no longer interpretable). Choose selection when you must explain the model, reduction when you only need fewer numbers. And selection isn't only about accuracy: weigh the operational cost of each feature — its serving latency, freshness requirements, compute cost, upstream data-dependency risk, and whether it's even available at prediction time. A feature that adds 0.1% AUC but depends on a flaky third-party call at serving time is usually not worth keeping. Finally, on redundancy: correlated features can actually *help stability* (keep both as insurance against drift), but genuine duplicates, proxies, and leaky features should still be removed.
Key points
- Use L1 regularization as your default feature selector for linear models — it zeros vestigial weights during training, giving you feature selection for free without a separate selection step. For the 500-feature fraud dataset: L1 on a logistic regression will drive most of the 450+ redundant or noisy features to exactly zero during a single training run. You get a sparse model with a built-in audit trail of which features are nonzero. No separate RFE pipeline needed.
- Trap: computing feature importance on the training set. Training-set importance reflects memorization; use a held-out validation set or permutation importance on the same evaluation data used for model selection. A tree trained on 500 features will assign high importance to features it memorized in training — including unique identifiers and near-duplicate features. Permutation importance on validation data measures whether shuffling the feature actually hurts predictive performance on unseen examples. These two rankings regularly disagree by large margins.
- Diagnostic: if removing the bottom 50% of features by importance hurts validation AUC by less than 0.5 points, those features were vestigial. If it hurts by more than 2 points, the importance ranking is likely wrong — the features are correlated and removing one changed others' apparent importance. This threshold test takes one additional evaluation run and tells you whether you have a clean selection or a collinearity problem. If the second case applies, switch from marginal importance ranking to permutation importance or SHAP values, which account for feature interactions.
- Selection uses the labels, so do it inside the CV folds and check stability. Ranking features on the full dataset before splitting leaks the held-out folds into the choice and inflates your score — run filter/wrapper/embedded/target-based selection inside each fold on the training portion only, in a Pipeline. And a one-run selection can be a fluke: stability selection reruns the choice across bootstraps/folds and keeps features chosen consistently (in ~90% of runs), which matters most with correlated features where the pick flips run to run (Lasso especially).
- Read the tools' caveats, separate selection from PCA, and weigh operational cost. Mutual information catches non-linearity but is univariate and sample-size-sensitive; SHAP splits credit among correlated features and shows model use, not causation; tree knobs (min_child_weight, gamma, colsample) are themselves regularised selection. Keep feature selection (subset of original, explainable features) separate from PCA (new uninterpretable components). And select on more than accuracy — serving latency, freshness, compute, data-dependency risk, and availability at prediction time all count; correlated features can aid stability, but duplicates/proxies/leaky features must go.
Feature selection is a bias-variance decision: too many features and the model memorizes noise; too few and it misses signal — and because correlation is not importance, the right ranking method matters as much as the threshold.
Recap
- Feature selection is a bias-variance decision: too many features memorize noise, too few miss signal — and correlation is not importance.
- L1 as the default selector for linear models: it zeros vestigial weights during training — selection for free, with a sparse audit trail.
- Importance on train reflects memorization (IDs, near-duplicates rank high) — use held-out permutation importance instead.
- Threshold test: dropping the bottom 50% costs <0.5 AUC → vestigial; >2 AUC → collinearity is fooling the ranking, switch to permutation/SHAP.
- Selection uses the labels — run it inside CV folds on the training portion, and check stability across bootstraps (Lasso picks flip run to run).
- Tool caveats: mutual info is univariate + sample-sensitive; SHAP splits credit among correlated features and shows use, not causation.
- Select on more than accuracy: serving latency, freshness, compute, data-dependency risk, availability — a flaky 0.1%-AUC feature isn't worth it.
Check your understanding
Q1. You filter features by Pearson correlation with target and retain only top 20. Your model performs worse than with all 100 features. What most likely went wrong?
- A) Retaining only 20 features likely dropped several highly correlated features that were providing redundant but stabilizing signal; ridge regression on all 100 would have been a better choice.
- B) The top-20 Pearson correlation cutoff was too aggressive — retaining the top 40 features instead would have preserved enough signal for good performance while still reducing overfitting risk.
- C) Pearson correlation measures linear dependence for a single feature IN ISOLATION and misses interactions and non-linear (U-shaped) relationships. Fix: mutual information or a wrapper method instead.
- D) The model overfit to the 20 selected features because the smaller feature set gave gradient descent fewer parameters to regularize, causing it to memorize the training data more aggressively.
Q2. Why does LASSO shrink some coefficients to exactly zero while Ridge (L2) rarely does?
- A) LASSO uses a higher default regularization strength than Ridge, causing more aggressive shrinkage; if Ridge were tuned to the same regularization strength as LASSO, it would also produce exact zeros.
- B) The constraint geometry differs: L2's penalty is a smooth sphere, so the optimum rarely lands on an axis. L1's penalty is a diamond whose corners sit on the axes, producing exact zeros there.
- C) LASSO uses coordinate descent optimization while Ridge uses gradient descent; coordinate descent naturally produces exact zeros as a numerical artifact of updating one coefficient at a time.
- D) LASSO applies the penalty to the raw coefficient values while Ridge applies it to the squared coefficients; squaring small values makes them even smaller, which paradoxically prevents Ridge from reaching zero.
Q3. You compute feature importance from a gradient boosted tree and find that 'customer_id' is the second most important feature. Which TWO of the following are true about what happened and the fix?
- A) Customer IDs encode temporal information — older customers have lower IDs and newer ones higher — so the model uses ID as a legitimate proxy for customer tenure, a genuinely predictive signal.
- B) Tree feature importance is biased toward high-cardinality features. customer_id is a unique identifier, so the tree can memorize training rows via it — huge importance at training time, zero generalization.
- C) The real fix is removing identifier columns before training, or using permutation importance instead, which measures actual held-out performance degradation rather than training-time impurity reduction.
- D) The feature-importance computation simply has a bug — customer_id should have been excluded from the matrix before training, and every other feature's score needs recomputing without it.
Q4. A linear model has two features that are 0.97 correlated (e.g., 'height_cm' and 'height_inches'). What specific failure mode does this cause and fix?
- A) Two near-identical features let the model assign a large weight to one and negative to the other — infinitely many combinations give the same prediction, so coefficients become wildly unstable.
- B) Two near-identical features cause the model to systematically double-count the effect of height, producing coefficients that come out at exactly half the true value for each of the two features.
- C) Near-perfect correlation between the two features causes gradient descent to oscillate noticeably during training, requiring a much smaller learning rate specifically to reach convergence reliably.
- D) Two features with 0.97 correlation will produce a VIF of exactly 16.9, computed as 1/(1-0.97 squared); since this sits below the common 50 threshold, intervention isn't strictly required here.
Q5. You rank all 500 features by mutual information with the target on the full dataset, keep the top 50, then run cross-validation and report the CV score. Why is that score optimistic?
- A) It genuinely isn't optimistic here — mutual information is fundamentally an unsupervised metric, so ranking features on the full dataset before splitting simply cannot leak anything.
- B) Feature selection used labels from the whole dataset, including rows that later become CV folds — those folds already helped pick the features. Run the ranking inside each fold instead.
- C) The reported score is optimistic only because keeping just 50 features is too few overall; expanding the kept set to 100 features would completely remove the underlying bias.
- D) Mutual information is simply the wrong metric to use here; switching to plain Pearson correlation while keeping the same overall procedure fully removes the optimism in the score.
Q6. A feature adds a genuine but tiny 0.1% AUC improvement, but computing it at serving time requires a call to a flaky third-party API with 300ms latency. A colleague insists on keeping it "because it helps." How should you frame the decision?
- A) Keep it unconditionally — any feature that measurably improves validation AUC, no matter how small the gain, should always be included in the final production model.
- B) Feature selection isn't only accuracy — weigh latency and dependency risk. A 0.1% gain with 300ms and a flaky call usually fails cost-benefit; drop it unless critical.
- C) Keep it, but cache the third-party API response for a full week, which entirely eliminates every dependency and freshness concern for this feature going forward.
- D) The decision here is purely statistical — if the 0.1% AUC gain tests significant at p less than 0.05, keep the feature; operational cost is simply not a modeling concern.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →