Online Learning and Concept Drift: When the World Changes Faster Than Your Model

Every production ML model degrades over time. User behaviour shifts, world events happen, seasonality cycles, and competitor actions change the distribution your model was trained on. Concept drift is the name for this. The question is not whether drift will happen but how fast you detect it, how you respond, and whether your system can learn continuously rather than in batches.

A model trained in January will not perform the same in December. User behaviour evolves. Product changes alter feature distributions. External events (pandemic, economic shock, viral trend) shift the entire world. Monitoring for and responding to this distribution shift is one of the core responsibilities of a production ML team.

Types of drift

Data drift (covariate shift): P(X) changes but P(Y|X) stays the same. The input distribution shifts — for example, a new user cohort has different demographic and behavioural patterns — but the relationship between features and outcomes is unchanged. A model trained on old data may underperform because it has not learned patterns relevant to the new distribution, but the model's learned function is still "correct."

Concept drift: P(Y|X) changes — the relationship between features and labels shifts. A fraud detection model trained on 2022 fraud patterns may miss entirely new fraud techniques in 2024. A sentiment model trained on pre-pandemic text may misclassify pandemic-era language. The model's learned function is now wrong, not just mis-calibrated.

Label drift: P(Y) changes. The overall class distribution shifts — for example, conversion rate drops from 5% to 2% without any feature change. Thresholds, decision rules, and calibration all need updating.

Detecting drift: statistical tests and monitoring

Population Stability Index (PSI): measures distributional shift in a single feature. PSI = Σ_{bins} (P_new - P_old) * ln(P_new / P_old). PSI < 0.1: negligible shift. 0.1–0.25: moderate shift, investigate. > 0.25: major shift, model retraining likely needed. Compute PSI for all input features and for model output scores.

Kolmogorov-Smirnov test: tests whether two samples come from the same distribution. Apply to each feature distribution separately. Chi-squared test for categorical features. These detect covariate shift. Detecting concept drift requires label data — comparing model predictions to ground truth labels over time. If the model's error rate increases and/or calibration degrades, concept drift is likely.

Monitoring architecture

Shadow mode: run the new model alongside the old model in production, serving the old model's decisions but logging the new model's predictions. Compare performance metrics before switching traffic. Canary deployment: route 1-5% of traffic to the new model, monitor key metrics, roll forward if stable, roll back if degraded. Champion-challenger: the current deployed model (champion) is continuously compared to candidate models (challengers) trained on more recent data. The challenger replaces the champion when its rolling performance exceeds the champion's for a sustained period.

Retraining strategies

Periodic retraining: retrain on a sliding window of recent data (e.g., last 90 days) on a fixed schedule (weekly, monthly). Simple to implement; does not respond to sudden drift. Event-triggered retraining: retrain when monitoring metrics drop below a threshold. Faster response; requires robust monitoring. Online learning: update model parameters continuously as new labelled data arrives (see FTRL for ads, Post 89). Fastest response but requires careful regularisation to prevent catastrophic forgetting.

Catastrophic forgetting: the online learning pitfall

When a model is updated continuously on new data, it may "forget" patterns from older data — a phenomenon called catastrophic forgetting. A fraud model updated on recent transactions may lose its ability to detect fraud patterns from six months ago that occasionally resurface. Mitigations: include a replay buffer of historical examples alongside recent data in each update; use elastic weight consolidation (EWC) to penalise large parameter changes from the previous model; monitor performance on a held-out historical test set.

Data quality monitoring: upstream of model monitoring

Models degrade when features degrade before the model sees them. Feature importance monitoring: track which features contribute most to model predictions. If a critical feature suddenly has 90% null rate (upstream pipeline failure), model performance may degrade without any change in the model itself. Schema validation: assert expected feature types, ranges, null rates, and cardinalities. Any deviation triggers an alert before the bad data reaches the model.

Try on Colab: simulate a classification dataset with concept drift. Train a model on the first 10,000 examples. Evaluate on examples 10,001-20,000 where the relationship P(Y|X) has shifted. Implement a sliding-window retraining: retrain monthly on the most recent 5,000 examples. Compare the static model vs sliding-window model AUC over time. Plot PSI for input features to detect the drift onset.