Concept Drift Detection

Page-Hinkley, ADWIN, and EMA checks for sudden, gradual, and recurrent drift

A credit model trained in 2019 is running through 2020 when the pandemic hits in March. Overnight, employment, spending, and default rates all shift. The model's features — employment status, recent spending, credit utilization — encode *pre-pandemic* relationships. After March, the same feature values mean something completely different: someone employed with steady spending is now a far worse risk than the old data would suggest, because the whole macro backdrop changed. Feature PSI reads 0.35 — severe. But even before PSI trips, the model's error rate on the post-March cohort is already compounding. This is sudden concept drift — the nastiest kind, because it doesn't build up gently enough to notice.

Three shapes of concept drift, three ways to catch them

*Sudden* drift is an abrupt regime change from a specific event — COVID, a new regulation, a competitor launch. It happens at a point in time. Catch it with the Page-Hinkley test (a running sum of errors crossing a threshold) or ADWIN, which compares a recent window to a historical one and collapses the window when they diverge. Both run on labeled errors, so "visible within days" means days after labels start arriving, not days after the drift event itself — for the credit-model example above with a 30-day label delay, that's within days of the 30-day mark; fast-labeled domains like fraud or ad clicks see the days-scale detection much closer to the event.

*Gradual* drift is a slow slide: an aging user base, evolving seasons, shifting preferences over months. No single day looks wrong. Catch it with an exponential moving average of the error rate that alerts when it stays above a control limit — for example, an error rate creeping from 2.0% to 2.6% over 90 days is a change of only +0.0067 percentage points a day, well inside normal daily noise, but that same 90-day window is exactly the sustained deviation an EMA control-limit check is designed to flag.

*Recurrent* drift is a pattern that returns: holiday fraud, annual cycles, weekday-vs-weekend behavior. Catch it with time-series decomposition of the error rate. The right response isn't retraining — it's keeping seasonal models and switching between them on schedule.

The hard constraint: detection lag

Concept drift is defined against *labels* — you can only truly confirm it once real outcomes arrive. A credit model with a 30-day label delay can drift undetected for a month; a fraud model with 7-day chargebacks, for a week. Feature PSI and score-distribution shifts are useful *leading* signals, but they fire on distribution change, not confirmed outcome change — treat them as prompts to investigate, not proof of drift.

Concept drift vs. its look-alikes. Concept drift is specifically P(Y|X) changing — the relationship between features and label breaking, which is what makes retraining a valid fix. Three things get confused with it: *data drift* is P(X) changing while P(Y|X) stays fixed (new email clients shift the feature distribution, but spam still looks like spam — recalibration, not retraining, is the fix); *prior shift* is P(Y) changing on its own (spam volume rises or falls, but the mapping from features to label is still correct — adjust the decision threshold to the new base rate, don't retrain); *infrastructure drift* isn't drift at all — a broken preprocessing step or extraction bug that mimics a performance drop and is fixed by rolling back the change, not retraining.

And don't reflexively retrain on recent data alone. Sudden drift usually does need retraining soon — the old patterns are dead for the affected segment — but that retraining should draw on a sliding window that includes enough pre-drift history, not just the most recent post-drift data, or you'll regress on the customers the regime change never touched (see the Trap below). Gradual drift can sometimes be handled with online updates or recalibration, and recurrent drift with seasonal model-switching. Full retraining is the most expensive lever, so exhaust cheaper recalibration and threshold adjustment first, and pull it only when they fail to close the gap.

Key points

• Set up an automated concept drift detection pipeline that monitors prediction accuracy on delayed labels — this is the ground truth signal. PSI and KS on features are early warning indicators that something may have changed. Model performance on actual labels is the definitive measure of whether that change matters. Run both: features give you the early warning before labels arrive, labels give you the confirmation that warrants retraining.
• Trap: retraining on only the most recent data after sudden drift. If pre-drift patterns still apply to a large portion of your user base — customers who were not affected by the regime change — a model trained only on post-drift data will regress on those users. Use a sliding window that includes enough pre-drift history to maintain performance across the full distribution. Validate on both pre-drift and post-drift held-out data before deploying.
• Diagnostic: after detecting drift, split validation data by time period — before and after the suspected drift date. If pre-drift accuracy is high and post-drift accuracy is low, you have confirmed the drift date. Retrain using a sliding window that includes pre-drift history — not post-drift data alone, which regresses the unaffected segment (see the Trap above) — and verify that the performance gap closes on both the pre-drift and post-drift validation slices. If it does not close after retraining, the features themselves may be insufficient to represent the new target relationship — feature engineering is required, not just data recency.

Recap

• Concept drift = world changed, model didn't: same features now mean something different.
• Sudden: abrupt regime change (COVID, regulation); catch with Page-Hinkley or ADWIN — visible in days.
• Gradual: slow slide over months; catch with an EMA of error rate crossing a control limit.
• Recurrent: returning pattern (holidays, seasons); catch via time-series decomposition — switch seasonal models, don't retrain.
• Detection lag is the hard constraint: confirmed only against labels (30-day credit, 7-day chargebacks).
• Don't retrain on only recent data: unaffected users regress; use a sliding window with pre-drift history.
• Exhaust recalibration first — full retraining is the most expensive lever, pulled only when cheaper fixes fail.

Check your understanding