Concept Drift: How to Detect It Before It Destroys Your Model
PSI > 0.2 — trigger alert. This is the rule every ML monitoring guide teaches. It's also frequently wrong for your specific use case. Here's a framework for actually understanding drift, choosing the right test, and setting thresholds that don't page you at 3am for seasonal fluctuations.
Drift comes in two flavors and you need to monitor for both differently.
Data drift (covariate shift): The distribution of input features X changes. P(X) ≠ P_ref(X). The model's learned mapping from X → y may no longer apply because it was learned on a different X distribution. Example: a model trained on desktop users now receives 40% mobile traffic.
Concept drift: The relationship between X and y changes. P(y|X) ≠ P_ref(y|X). Even if features look the same, the world has changed. Example: a price sensitivity model trained pre-inflation now misprices customers because the same income bracket has different purchasing behaviour.
Detection methods:
PSI (Population Stability Index): Bins the distribution, computes Σ (actual% - expected%) × ln(actual%/expected%). Good for univariate feature monitoring. Standard thresholds: < 0.1 stable, 0.1–0.2 monitor, > 0.2 significant. But: binning strategy matters enormously. Equal-width bins will miss tail drift. Use quantile-based bins for heavy-tailed distributions.
KS test: Kolmogorov-Smirnov measures the maximum absolute difference between CDFs. Doesn't require binning. More sensitive to tail shifts than PSI. Use for continuous features where tail behaviour matters (revenue, session duration).
Model-based drift: Train a classifier to distinguish reference vs current data. If it achieves AUC > 0.7, drift is detectable. This catches multivariate drift that univariate tests miss.
Setting thresholds that work:
The 0.2 PSI threshold assumes your reference distribution is stable. If your data has weekly seasonality (as most consumer data does), Monday PSI vs Saturday will always be > 0.2 without any drift. Options: (1) Use same-weekday comparison as reference. (2) Apply seasonal decomposition before computing PSI. (3) Use rolling window comparison (last 7 days vs prior 7 days) instead of fixed reference.
Distinguishing data drift from concept drift in production:
You can observe data drift immediately (compare feature distributions). Concept drift requires labels, which often come with a lag. Bridge: use proxy metrics. For a revenue model, track predicted vs actual revenue. For a ranking model, track predicted CTR vs observed CTR on the same items. Divergence = concept drift signal, no labels required.
```python import numpy as np
def compute_psi(expected, actual, n_bins=10): """PSI with quantile-based bins — avoids the equal-width trap on skewed data.""" bins = np.percentile(expected, np.linspace(0, 100, n_bins + 1)) bins[0], bins[-1] = -np.inf, np.inf # open-ended edges
exp_cnt = np.histogram(expected, bins=bins)[0] act_cnt = np.histogram(actual, bins=bins)[0]
exp_pct = np.where(exp_cnt == 0, 0.001, exp_cnt / len(expected)) act_pct = np.where(act_cnt == 0, 0.001, act_cnt / len(actual))
psi = np.sum((act_pct - exp_pct) * np.log(act_pct / exp_pct)) return psi
# Production usage psi = compute_psi(reference_scores, production_scores) label = ("Stable" if psi < 0.1 else "Monitor closely" if psi < 0.2 else "Significant drift — page on-call") print(f"PSI={psi:.3f} → {label}")
# Tip: use same-weekday reference to avoid false positives on seasonal data ```