Why Training-Serving Skew Silently Kills Production Models
Your model has 0.92 AUC in the notebook. It degrades to baseline within two weeks in production. Nobody's alarmed because the metrics move slowly. Then one day someone checks the actual conversion rate and it's half of what it was at launch. The culprit is almost always the same thing: a gap between how you compute features at training time and how you compute them at serving time.
The gap between a model that works in a notebook and one that works in production is almost always a feature engineering problem.
The four most common forms of training-serving skew:
1. Timestamp boundary bugs. Your training pipeline computes a 7-day rolling window with `WHERE ts < event_ts`. Your serving code uses `WHERE ts <= NOW() - INTERVAL 7 DAYS`. On the surface these look the same. In practice, training sees a half-open interval at each event's timestamp, while serving uses wall-clock time — and at midnight, they diverge by up to 24 hours.
2. Different null handling. Training imputes missing `age` with the column mean (34.2). Serving code was written by a different team six months later, and they imputed with 0 for simplicity. A 0-age user looks like a child to your model. Every user with a missing age now gets systematically mis-scored.
3. Scaler fitted on wrong data. The classic. `scaler.fit_transform(X_train)` at training time. `scaler.fit_transform([single_row])` at serving time — re-fitting on every request. A single row has mean = its own value, std ≈ 0. Every scaled feature becomes 0 or undefined.
4. Aggregation window timezone mismatch. Training uses a calendar-day window (midnight UTC). Serving uses a rolling 24h window. At 11pm UTC, serving might include the next calendar day. Small difference, consistent bias across all temporal features.
How to detect it:
Log your serving features — not just predictions. Ship a feature logging sidecar to your serving infrastructure. Every hour, run PSI between the logged feature distribution and the training distribution baseline. PSI > 0.1: investigate. PSI > 0.2: incident.
How to prevent it:
Use a feature store. A real one. The same Python code that computes features for training should be the same code that computes them at serving — called through the same interface. The feature store doesn't need to be fancy. Even a shared library with unit tests is better than two separate code paths maintained by different teams.
The engineering cost of a feature store is front-loaded. The cost of not having one compounds indefinitely.