Model Explainability: SHAP, Permutation Importance, Local vs Global

Stakeholders ask "why did the model predict this?" Regulators demand "what features influenced this decision?" Engineers debug "why is this feature suddenly the top one?" These are three different explainability questions with three different right tools. Getting them confused is one of the most common failures in production ML.

Model explainability is a layered subject. There is no single "explain the model" tool — there are several techniques that answer different questions, and using the wrong one leads to wrong conclusions. The senior practitioner knows which tool answers which question and which traps each tool has.

Local vs global explanations

The first distinction. Global explanation tells you what the model has learned overall: which features matter on average, across all predictions. Local explanation tells you what drove a specific individual prediction: which features mattered for this user, this transaction, this query.

Stakeholders and regulators usually want global ("which features does this credit model rely on?"). Operations and customer support usually want local ("why was this specific customer denied a loan?"). The two are not interchangeable. A feature can have high global importance and low importance for any specific prediction (when the feature interacts strongly with others). A feature can have high local importance for an individual case and low global importance (when it is rarely informative but very informative when it is).

Coefficient inspection (linear models)

For linear regression and logistic regression, the model's weights are the explanation. The magnitude (after standardising features) tells you global importance. The sign tells you direction. The actual prediction for any input is a linear combination, fully transparent.

This is why linear models remain the workhorse of regulated ML. A credit model where every weight can be inspected and explained to a regulator is fundamentally easier to ship into a regulated industry than a neural network with sixty million parameters, even if the neural network is more accurate.

Tree-based feature importance

For decision trees, Random Forests, and gradient boosted models, the standard feature importance is "gain" — how much each feature reduced the loss on average across all the splits where it was used. This is fast to compute, always available, and biased.

The bias: gain-based importance favours high-cardinality features (features with many possible split points). user_id, session_id, timestamp — these often appear as top features even when they're not genuinely informative, because they have many possible splits and any residual variance the regularisation didn't squeeze out gets attributed to them. Acting on gain-based importance ("we should focus on user_id as a feature") is how teams waste sprints on the wrong signal.

Permutation importance

The honest alternative. For each feature, randomly shuffle its values and measure how much the model's performance drops on a held-out set. Big drop means the feature was important; small drop means it was not. The advantage over gain: not biased by cardinality. The cost: slow to compute (one model evaluation per shuffle per feature) and sensitive to feature correlations (when two features are correlated, shuffling one doesn't hurt accuracy because the model uses the other; both features end up looking unimportant).

The standard fix for the correlation problem is grouped permutation importance: shuffle correlated features together as a block. This costs more analysis upfront but gives a more honest measure.

SHAP (Shapley Additive exPlanations)

The most principled local explanation method. For each prediction, SHAP assigns a value to each feature representing its marginal contribution to the prediction, averaged over all possible orderings of features. SHAP values sum to the difference between the prediction and a baseline (the model's expected output).

The advantages: SHAP values are theoretically grounded (Shapley values from game theory), they satisfy desirable properties (efficiency, symmetry, dummy, linearity), and they give per-prediction explanations. SHAP can also be aggregated across many predictions to get a global view (mean absolute SHAP per feature), which gives a more honest global importance than gain or permutation on the same data.

The cost: SHAP is computationally expensive. For tree models, TreeSHAP computes exact SHAP values efficiently. For neural networks, you need DeepSHAP or sampling-based approaches and the quality of the explanation degrades. For LLM outputs, SHAP is essentially intractable; you have to use other techniques (attention maps, perturbation-based interpretability).

The interpretation traps

SHAP explains the model's behaviour, not the real world. A feature with high SHAP value is one the model is using, which is not the same as a feature that has a causal effect on the outcome. If the model learned to use a proxy for the true cause, SHAP will tell you the proxy is important — which is true about the model but not about the world.

Permutation importance assumes feature independence. With correlated features, individual permutation importance under-states each feature's true importance (because the model fills in from the correlated partner). Grouped permutation or conditional permutation are the fixes.

Gain-based importance assumes the model's training-time information is the truth. When features are highly leakage-prone (computed from post-event aggregates, for example), gain might be high simply because the leakage was useful at training time. The feature is not "important" in any production-meaningful sense.

WARNING — Production tell: SHAP says this feature is the top driver; removing it doesn't change the model's predictions much. What happened? Almost always: correlated features. The "top driver" feature has high SHAP because the model used it heavily in training; but a near-duplicate feature is also available and absorbs the signal when the top one is removed. The model is robust to losing any single feature in a correlated group while looking like it depends heavily on each one. The senior move: cluster features by correlation first, then assess importance at the cluster level, not the feature level. This is also why SHAP-based feature engineering ("drop low-SHAP features") can fail badly — you might drop a feature that was the only reliable carrier of an important signal.

When explainability matters

Three production scenarios where this is non-negotiable. (1) Regulated industries — credit scoring, medical diagnosis, insurance underwriting — where the model's decisions must be explainable to a regulator or a customer. (2) Debugging — when the model produces a surprising prediction and you need to know why before you can decide whether to trust it. (3) Trust and adoption — when stakeholders need to understand the model to incorporate its outputs into their decisions. In each case, the choice of explanation method has to match the question being asked.

Interview questions on this topic

"A regulator asks you to explain why the credit model denied this customer. What do you give them?" — A local explanation. SHAP values for the individual prediction, showing which features pushed the model toward "deny" and by how much. Plus a brief sentence per top feature describing what it represents. Not a global importance chart — the regulator asked about a specific customer, not the model in general. Not the model coefficients alone — modern tree models do not have a single coefficient per feature. The right answer is per-prediction SHAP rendered as a force plot or a waterfall chart, with English descriptions.

"What is the difference between 'this feature is important' and 'this feature is causally important'?" — The former is a statement about the model; the latter is a statement about the world. A model can learn to use a proxy (e.g., zip code) that correlates with the true causal driver (e.g., neighbourhood quality, income). The feature is important to the model but not causally important — interventions on the feature would not change the outcome. Explainability methods (SHAP, permutation) tell you about the model's behaviour, not about causal effects. For causal claims, you need causal inference methods (post 47, 84, 85 in Gradient).

"Permutation importance says all of your top 5 features are equally important. Is this useful?" — Only if you check for feature correlations. If the top 5 features are highly correlated, each one has low permutation importance individually because the model can substitute. The conclusion "they are equally important" is misleading — they might all be redundant carriers of the same underlying signal. Grouped permutation importance, or correlation-aware methods, give a more honest picture. Equal individual permutation importance with high inter-correlation is a sign that the model has heavy feature redundancy.

"You see a feature with high gain-based importance but low SHAP-based importance. What's likely happening?" — Gain over-weights features that get used in many splits — typically high-cardinality features like user_id or timestamps. SHAP gives a more honest picture of marginal contribution. The feature is probably being used as a high-cardinality regulariser (the model splits on it to memorise specific examples) rather than as a genuinely informative feature. Drop it, retrain, see if anything changes — usually nothing does. This is one of the most reliable smoke tests for feature usefulness.

Try on Colab: train an XGBoost model on a tabular dataset (Adult Income, German Credit, any reasonable choice). Compute (1) gain-based feature importance, (2) permutation importance on a held-out set, (3) mean absolute SHAP values using TreeSHAP, (4) for a specific individual prediction, the per-feature SHAP values. Compare the three global importance rankings — they will disagree. Investigate the disagreements: which features have high gain but low SHAP, which have high permutation but low SHAP, what does that tell you about how the model is using them. This is the practical experience of explainability in production.