Data Drift Detection

PSI, KS test, chi-squared, Jensen-Shannon divergence, choosing thresholds

A loan model goes live in January. By March a new kind of applicant is showing up — recent graduates, a newly opened geographic market — and their median income has moved from about 55K to 70K dollars. The model still runs, still returns predictions at the same speed, still fires no alerts. But it is quietly making worse decisions, because it was tuned for a population that no longer shows up at the same rate. By the time a stakeholder notices the approval or default rate has drifted, it's July — and the root cause is six months old. This is silent model decay, and it is the *default* outcome when nobody is monitoring.

The idea: watch the inputs, don't wait for the labels

Instead of waiting for outcomes to confirm the damage, you compare each input feature's *current* distribution against its *training* distribution and measure how far apart they've moved. The question becomes concrete: how much has the income distribution shifted, and is that shift big enough to matter?

PSI: the metric the lending world built for exactly this

Population Stability Index sums, across bins of a feature, how much probability mass moved:

$PSI = \sum (p_{train} - p_{new}) ln(p_{train}/p_{new})$

Below 0.1 the population is stable; 0.1–0.2 is mild drift worth a look; above 0.2 is real drift that needs action. For income moving from 55K to 70K, here's the arithmetic behind that claim: split training income into 5 equal-frequency bins (<35K, 35–50K, 50–65K, 65–80K, >80K), each holding 20% of the training population by construction. After the shift, those same bins hold roughly 8%, 12%, 18%, 32%, and 30% of the new population — mass moves out of the bottom bins into the top two. Summing $(p_{train}-p_{new})ln(p_{train}/p_{new})$ bin by bin gives 0.110 + 0.041 + 0.002 + 0.056 + 0.041 ≈ 0.25, above the 0.2 action threshold — an actionable signal *months* before any label confirms the harm.

Other tests for other shapes of data

For continuous features, the Kolmogorov-Smirnov test takes the largest gap between two cumulative distributions:

$D = max|F_{train}(x) - F_{new}(x)|$

It's distribution-free and notices shifts anywhere, not just in the mean. The trap: with a million daily requests, D = 0.015 will be "statistically significant" (p < 0.001) yet operationally meaningless — so pair it with a *practical* floor like D > 0.05. For categorical features, chi-squared checks whether category frequencies still match training: $chi^2 = sum rac{(O_i - E_i)^2}{E_i}$, comparing observed category counts $O_i$ against the counts $E_i$ expected under the training distribution. And when you want one bounded, uniformly-thresholdable number across very different features, Jensen-Shannon divergence — computed with log base 2, unlike the natural-log PSI and KS above — sits neatly in [0, 1].

One caution on what an alert means. Statistical significance is not business significance. A drift alert means *investigate,* not *retrain.* Sometimes a feature moves a lot and the model stays accurate because the learned relationship still holds; other times a subtle shift in one important feature wrecks accuracy while 49 others look fine. So you run both: drift detection as the early warning that *something* changed, and performance monitoring on delayed labels as the confirmation that it *matters.* The alert says "look here." The labels tell you if it's a real problem.

Key points

• Use PSI for binned continuous and ordinal features as your default drift metric. Build 10 equal-frequency bins from the training distribution — not equal-width, because skewed distributions put all the signal in a few dense center buckets with equal-width binning. Add boundary bins for values falling outside the training range. Apply the rule: PSI < 0.1 is stable, 0.1–0.2 is investigate, > 0.2 is act. The loan income example (5 equal-frequency training bins at 20% each shifting to roughly 8/12/18/32/30% after the move to 70K) computes to PSI ≈ 0.25 — see the worked bin-by-bin sum above, not just an asserted number. These thresholds are stable enough to apply without per-feature recalibration, which is why the financial industry standardized on them.
• The most common production trap is acting on statistical significance rather than practical significance. With 1M daily serving requests, the KS test will flag D = 0.015 as p < 0.001. That is a shift of 1.5 percentage points in the CDF — real, but almost certainly not affecting model performance. Engineers who fire a retraining pipeline on every statistically significant drift alert spend all their time on retraining overhead and still miss the actual incidents, because the threshold is too sensitive. Set D > 0.05 as your practical floor. For PSI, trust the 0.1/0.2 boundaries — they were empirically calibrated over decades of financial model deployment, not derived from theory.
• The diagnostic: monitor prediction score distribution first, then feature distributions. The prediction score distribution changes before any feature drift alerts fire and before any labels arrive. A loan model's score distribution shifting from mean 0.35 to mean 0.28 over two months is the earliest signal — the model is scoring the new population differently. Once you see score drift, run PSI on each feature ordered by training-time importance. The first high-PSI, high-importance feature is your root cause. This narrows a 50-feature investigation to a 1-feature investigation within minutes.

Recap

• Watch inputs, don't wait for labels: silent decay is the default when nobody monitors.
• PSI = default drift metric: $PSI = \sum (p_{train} - p_{new}) \ln(p_{train}/p_{new})$; <0.1 stable, 0.1–0.2 investigate, >0.2 act.
• Use 10 equal-frequency bins, not equal-width — skew hides signal in dense center buckets.
• KS for continuous: $D = \max|F_{train}(x) - F_{new}(x)|$; distribution-free but pair with a practical floor $D > 0.05$.
• Significance ≠ business significance: 1M requests make $D=0.015$ "significant" yet meaningless.
• Score distribution shifts first, before feature alerts or labels — then run PSI ordered by feature importance.
• A drift alert means investigate, not retrain — only labels confirm the drift actually matters.

Check your understanding