Feature Importance Drift

SHAP drift, permutation importance over time, what it reveals

A fraud model went live six months ago. On day one its top features were `transaction_velocity` (0.32), `device_age` (0.21), `ip_reputation` (0.18). Six months later the order has flipped: `ip_reputation` (0.38), `transaction_velocity` (0.12), `device_age` (0.09). IP reputation has quietly become the model's dominant signal. Why?

The cause is the model's own side of the world

The fraud team spent those six months aggressively blocking high-risk IPs. So the fraud that *survived* now comes from addresses that were never flagged — the leftover fraud has low-risk IP reputation. The model dutifully learned that "low-reputation IP" is now the reliable tell, because that's what surviving fraud looks like. The catch: it's now leaning on a signal that its own upstream actions made gameable, which makes it far more exposed to IP spoofing. The model adapted to a world that its own team reshaped.

Why this is worth monitoring at all

Feature importance drift isn't just "what the model is doing" — it's a read on *how the world moved relative to the model's assumptions.* Four things drive it: a feature's distribution narrows, shrinking its discriminative power; the target population shifts (from the model's own actions or outside forces), changing which examples remain; a feature's quality degrades from a pipeline bug, going noisy or partly null; or a brand-new correlation appears as behavior changes.

To watch it, compute SHAP or permutation importance on a rolling sample of ~1,000 production predictions per week, track the top-K over time, and alert when a feature's rank moves more than 3 places or its importance changes by more than 20%.

The mental correction: importance is *not* a fixed property of the model. It's a joint property of the model and the input distribution — the same weights yield different importances when the inputs shift. That's what makes it such a sensitive probe: a pipeline bug that corrupts one feature shows up in importance drift within days, well before it has piled up enough label evidence to move the accuracy metric.

Key points

• Monitor SHAP-based feature importance weekly on a rolling production sample — it catches feature pipeline degradation and concept drift earlier than any accuracy-based metric. A feature that drops from rank 2 to rank 15 in one week has either lost its signal (distribution collapsed) or its pipeline broke. Either way, you know where to look before accuracy confirms the damage.
• Trap: using training-time feature importance as the production reference. Training importance reflects the training distribution. Compute production importance from actual production traffic and compare to your deployment-day baseline, not training-day baseline. The deployment-day baseline is your "healthy production" reference — it captures the live feature distribution, not the historical training distribution.
• Diagnostic: if importance shifts without a corresponding drift in that feature's own distribution, the target-feature relationship has changed — this is concept drift. If importance shifts alongside distribution drift for that feature, the feature pipeline is the likely cause. The distinction determines whether you retrain (concept drift) or fix the pipeline (infrastructure drift). Running both feature distribution monitoring and importance monitoring in parallel gives you this diagnosis within one week.

Recap

• Importance is joint, not fixed: a property of model AND input distribution, so it shifts when inputs shift.
• Rank flips reveal how the world moved: `ip_reputation` rose to #1 because blocking left only low-reputation fraud.
• Four drivers: narrowing distribution, population shift, pipeline degradation, or a new correlation.
• Compute SHAP/permutation on ~1,000 rolling predictions/week; alert on rank move >3 or importance change >20%.
• Baseline against deployment-day, not training-day — training importance reflects the wrong distribution.
• Shift without distribution drift = concept drift; shift with it = pipeline — run both to get the diagnosis.
• Catches pipeline bugs in days, before enough label evidence moves accuracy.

Check your understanding