SHAP Values: Feature Importance That Actually Makes Sense

Feature importance from a random forest is not the same as feature contribution to a specific prediction. And feature contribution is not the same as causal effect. SHAP values give you the first correctly — and they're game-theoretic, not heuristic. Here's what that actually means.

There are four common ways to compute feature importance, and they measure four different things.

1. Impurity-based importance (default in sklearn forests): How much did this feature reduce impurity (Gini/entropy) on average across all splits? Problem: biased toward high-cardinality features. A random UUID feature will have high importance by this measure. Don't use this for anything serious.

2. Permutation importance: Randomly shuffle one feature's values, measure the drop in model performance. Measures how much the model relies on this feature. Not biased by cardinality. Slow for many features. Can be misleading when features are correlated — shuffling one correlated feature still leaves signal in the others.

3. LIME (Local Interpretable Model-agnostic Explanations): Fit a simple linear model around a specific prediction in the local neighbourhood. Fast, intuitive, model-agnostic. Problem: the neighbourhood is defined heuristically, explanations are unstable, and they don't guarantee consistency between local explanations.

4. SHAP (SHapley Additive exPlanations): Based on Shapley values from cooperative game theory. The Shapley value of player i is the average marginal contribution of i across all possible orderings of players.

In ML terms: the SHAP value for feature j on prediction x is the expected change in model output when we add feature j to a coalition, averaged over all possible feature coalitions. It satisfies three axioms: efficiency (SHAP values sum to prediction minus baseline), symmetry (features with equal contributions get equal values), dummy (features with no effect get zero SHAP value).

TreeSHAP: For tree-based models (XGBoost, LightGBM, sklearn forests), SHAP can be computed exactly in O(TLD²) where T=trees, L=leaves, D=depth. This is fast enough for production feature attribution.

What SHAP tells you — and what it doesn't:

SHAP measures model-based contribution — not causal effect. A feature can have high SHAP values because it's correlated with a causal feature that isn't in the model. Don't use SHAP to make causal claims.

Useful applications: explaining individual predictions to fraud analysts, auditing model behaviour for protected attributes, debugging why a model makes unexpected predictions, prioritising feature development.

Common misuses:

Using global SHAP importance (mean |SHAP|) as the only ranking — misses local heterogeneity. Showing SHAP plots without a baseline — the "expected output" interpretation only makes sense with a reference distribution. Using SHAP to justify removing features — high SHAP value can come from a feature correlated with many others; removing it may or may not hurt performance.