Three Drift Signals That Predict Model Failure Before It Happens

By the time a model failure shows up in your business metrics, it's been degrading for weeks. The signals were there earlier — in your input distributions, your prediction distributions, and your residuals — but nobody was watching. Here's what to monitor, what thresholds matter, and what each signal actually tells you.

Model failures don't announce themselves. They compound quietly across days or weeks until a stakeholder notices conversion is down 18% and you spend a week reverse-engineering what happened. The information was available the whole time — just not in the right place, with the right alert.

There are three layers of drift to monitor. They detect different failure modes at different points in the degradation timeline.

Layer 1: Input drift — the earliest signal

Input drift means your features are moving away from the training distribution. This doesn't guarantee model performance has degraded yet, but it's a leading indicator — the model is now operating in territory it didn't train on.

The standard metric is PSI (Population Stability Index). For each feature:

PSI is computed by binning the reference distribution (training) and current distribution, then summing: `Σ (Actual% - Expected%) × ln(Actual% / Expected%)`.

For categorical features, track each category's share separately. The most common trigger: a new categorical value appears in production that was never seen during training. Your model's embedding for that value is random or zero — it will mispredict for every row with that category.

Monitor input drift daily for high-cardinality features, weekly for stable ones. Alert on PSI > 0.15 for the top 10 features by importance.

Layer 2: Prediction drift — the performance proxy

You can't always get ground truth labels in real time (it takes 30 days to know if a loan defaulted). But you can watch how the model's predictions are distributed.

Prediction drift compares the distribution of P(Y=1) today vs at the time of deployment. Use the KS (Kolmogorov-Smirnov) statistic: the maximum absolute difference between the two CDFs.

A healthy model: KS < 0.05. Its predictions today look like its predictions at launch.

A drifting model: the mean prediction drops from 0.23 to 0.18 over two weeks, and the KS grows to 0.14. The model is becoming more conservative — possibly due to feature drift, or due to a real shift in the population.

Critical distinction: prediction drift can be a false alarm. If the underlying base rate genuinely changed (fewer people are actually likely to default this month), you want the predictions to shift. The signal is "investigate," not "rollback." Compare prediction drift against any known base rate changes.

Layer 3: Residual drift — the ground truth signal

When labels are available (even with delay), compute model residuals for a rolling cohort: residual = y_true - y_pred (for regression) or look at calibration curves (for classification).

Systematic residual drift — where the model consistently over- or under-predicts a specific segment — indicates concept drift: the relationship between features and labels has changed. This is the hardest drift to catch early but the most actionable.

For binary classification: run calibration checks by cohort. If your model predicts 0.3 probability but the actual rate for that score bucket is now 0.45, the model is miscalibrated and needs either a calibration update or a full retrain.

The monitoring architecture that works

You don't need an expensive MLOps platform for this. You need:

1. Log serving features alongside predictions (not just predictions alone) 2. Compute PSI daily vs training baseline — a 30-line SQL query 3. Compute prediction KS weekly vs launch week distribution 4. When labels arrive: compute residuals by score decile, alert if calibration error > 0.05

The teams that catch model failures early aren't the ones with the most sophisticated tooling. They're the ones who consistently run these three checks and act on the alerts rather than explaining them away.

```python import numpy as np import pandas as pd from scipy.stats import ks_2samp

def compute_psi(expected, actual, buckets=10): """Population Stability Index — PSI > 0.2 means retrain.""" breakpoints = np.percentile(expected, np.linspace(0, 100, buckets + 1)) breakpoints[0] = -np.inf breakpoints[-1] = np.inf

exp_counts = np.histogram(expected, bins=breakpoints)[0] / len(expected) act_counts = np.histogram(actual, bins=breakpoints)[0] / len(actual)

# Clip to avoid log(0) exp_counts = np.clip(exp_counts, 1e-6, None) act_counts = np.clip(act_counts, 1e-6, None)

psi = np.sum((act_counts - exp_counts) * np.log(act_counts / exp_counts)) return psi

# Usage: compare training distribution vs last 7 days of production psi = compute_psi(train_feature_values, prod_feature_values_last_7d) print(f"PSI: {psi:.3f} — {'STABLE' if psi < 0.1 else 'MONITOR' if psi < 0.2 else 'RETRAIN'}")

# KS test for distribution shift ks_stat, p_value = ks_2samp(train_feature_values, prod_feature_values_last_7d) print(f"KS p-value: {p_value:.4f} — {'SHIFT DETECTED' if p_value < 0.05 else 'stable'}") ```