Concept Drift: The Invisible Enemy That Stalks Every Production Model
PSI looks clean. Feature distributions look stable. Your model is quietly wrong. Concept drift — where the relationship between inputs and outputs changes — is undetectable by feature monitoring alone and is the leading cause of silent model degradation.
Three Types of Drift — and Why Only One Is Fatal
Drift is overloaded. When engineers say "drift," they usually mean input distribution shift: the features your model receives in production look different from features it was trained on. This is measurable, monitorable, and often recoverable. But it is not the dangerous kind.
Input distribution drift (covariate shift): P(X) changes. Your age distribution shifts. A new market segment appears. A feature pipeline changes encoding. PSI detects this. Retraining on recent data fixes it.
Label distribution drift: P(Y) changes. Your fraud rate doubles due to a new attack vector. Class imbalance in production diverges from training. This is detectable by monitoring outcome rates, but only if you have outcomes — which requires label delay tolerance.
Concept drift: P(Y|X) changes. The relationship between features and labels changes, even though the features themselves look the same. This is the dangerous kind. PSI cannot detect it. Feature histograms look fine. Your model was trained correctly on historical data — the problem is that the world has changed in a way that makes that history misleading.
The Credit Model Trained Pre-Pandemic
A credit risk model trained in 2019 on pre-pandemic consumer behavior was deployed into 2020. Income features looked similar. Employment features looked similar. Debt-to-income ratios were in-distribution. But the relationship between these features and default probability had fundamentally changed. Consumers who would have defaulted given their feature profile in 2019 were being kept afloat by stimulus payments. Consumers who looked safe by historical standards were being hit by sector-specific unemployment. PSI on individual features showed nothing alarming. The model's predictions were quietly wrong.
This is concept drift. The inputs are stable. The world has changed. The mapping from inputs to outcomes no longer matches what the model learned.
Why PSI Cannot Help You Here
PSI (Population Stability Index) compares the distribution of a feature or score between two time windows. It tells you when P(X) has shifted. It tells you nothing about P(Y|X). You can have PSI=0 on every feature and still have severe concept drift if the world has changed in ways that leave your feature distributions intact while invalidating the relationships between them.
The engineers who rely on PSI alone as their drift monitor are flying blind for the most important type of drift.
Detection: What Actually Works
Prediction distribution monitoring: track the distribution of your model's output scores over time. If the score distribution changes, something has changed — either in inputs or in the model's behavior on them.
Outcome rate tracking: compare your model's predicted positive rate to the actual observed positive rate (with appropriate label delay). If your model predicts 8% fraud rate but you are observing 15%, your model is underestimating risk — either due to concept drift or label distribution shift.
Residual drift: for regression models, track the distribution of residuals (predicted minus actual) over time. If residual mean drifts from zero, your model has become systematically biased in a specific direction.
Retrain vs Recalibrate vs Rollback
If prediction distribution has drifted but outcomes are stable: recalibrate. The model's rankings may still be correct; only its probability estimates have shifted.
If outcome rate has diverged from model scores significantly: retrain on recent labeled data. The mapping P(Y|X) has changed and the model needs to relearn it.
If performance has degraded catastrophically in a short window (days, not months): investigate before retraining. A sudden collapse suggests a data pipeline bug, not drift. Retraining on corrupted data will propagate the corruption.
Rollback is the right call when: (1) a recent deployment is the suspected cause, (2) the performance regression is acute, and (3) you have a known-good prior model to fall back to. Rollback buys time for root-cause analysis — it is not a fix.