Training-Serving Skew: The Complete Taxonomy and Detection Framework
Your model has 0.91 AUC in the notebook. Two weeks in production, it degrades to baseline. Nobody changed the model. Nobody changed the data. The gap is training-serving skew — a systematic difference between how features are computed at training time and how they're computed at serving time. It's the single most common source of production ML failure, and it hides in four distinct forms that each require different prevention strategies.
Training-serving skew is the gap between model quality in an offline evaluation environment and model quality in a production serving environment. The model is the same. The code that computes features at training time and serving time is different — and that difference is what breaks the model.
Why this happens systematically:
Training happens once. An engineer (or a team) writes a feature computation pipeline. It runs on historical data, generates training examples, the model trains and validates offline, and metrics look good.
Serving happens continuously. A different engineer (or a different team, or the same engineer months later without context) writes serving code that computes features at prediction time. They have incomplete information about the training pipeline. They make different assumptions. The two code paths diverge.
The model has learned from training features. Production features look different. Predictions degrade.
The four canonical forms of training-serving skew:
1. Timestamp boundary bugs
Training computes a 7-day rolling window with `WHERE timestamp <= event_ts`. Serving computes a 7-day rolling window with `WHERE timestamp <= NOW() - INTERVAL 7 DAYS`. Both are "7 days" — they should be identical.
They're not. At midnight UTC, they diverge by up to 24 hours. A training event at 08:00 UTC gets a 7-day window from [T-7d 08:00, T 08:00]. A serving request at 08:05 UTC gets a window from [NOW-7d 00:00, NOW 00:00]. Boundary mismatch → feature value difference.
Production signal: features shift noticeably at midnight UTC. Your model's prediction for the same user differs by a few percentage points depending on when the request arrives relative to UTC midnight.
2. Different null handling
Training imputes missing `customer_age` with the column mean (34 years). Serving code uses 0 for missing values "for simplicity." A user with missing age gets 34 in training (looks like an adult) and 0 in serving (looks like invalid/new user). The model has learned separate patterns for age 0 and age 34 — they are different in its decision boundary.
Production signal: new users and edge cases (null demographic fields) perform worse than average. A slice analysis shows precision drops 15% for users where a demographic field is missing.
3. Scaler fitted on wrong data
Training: `scaler.fit(X_train)` then `scaler.transform(X_train)` and `scaler.transform(X_val)`. All features are standardised to zero mean and unit variance across the training distribution.
Serving: `scaler.fit_transform([single_row])` every request. A single row has mean = itself and std ≈ 0. Every scaled feature becomes near-zero or NaN. The model receives a feature vector that it has never seen: all zeros.
Production signal: scaled features disappear (become NaN or 0). Model performance is random. This is the most catastrophic form.
4. Aggregation window timezone or calendar mismatch
Training: "daily active users" computed as users active between [calendar_day_start_UTC, calendar_day_end_UTC]. A user active on Dec 31 at 23:55 UTC and Jan 1 at 00:05 UTC is counted as 2 calendar days.
Serving: "daily active users" computed as a rolling 24-hour window from wall-clock time. Same user is counted in a single 24-hour window.
Small difference per user. Consistent bias across aggregate features. The model learns patterns from calendar-day aggregation; production receives rolling-window aggregation.
Detection framework:
Step 1: Log serving features. Alongside every prediction, log the feature vector. Use the same precision and serialisation as training. Send to a central store.
Step 2: Compute daily PSI. For each feature: compute PSI(training_distribution, serving_distribution_from_logs). PSI > 0.1: investigate.
Step 3: Identify features with unexplained drift. Features where PSI is high but you haven't deployed changes. Compare the feature computation code between training and serving. Document differences.
Step 4: Implement a feature parity test. Take 100 historical events. Compute features for each event using the training pipeline AND the serving pipeline (retroactively, with the same data the serving pipeline would have received at that time). Compare. Any feature with mean difference > 0.05 on a normalised scale is suspect.
Step 5: Establish a feature store baseline. Use a feature store (Feast, Tecton, Hopsworks) that enforces identical code paths for training and serving. Invest the time upfront; it pays for itself in prevented incidents.
```python import pandas as pd import numpy as np
def detect_training_serving_skew(train_features: pd.DataFrame, serving_log: pd.DataFrame, threshold: float = 0.1) -> dict: """ Compare training feature distributions vs logged serving features. Returns per-feature PSI. PSI > 0.1 = investigate. PSI > 0.2 = incident. """ results = {} common_cols = set(train_features.columns) & set(serving_log.columns)
for col in common_cols: train_vals = train_features[col].dropna().values serve_vals = serving_log[col].dropna().values
if len(serve_vals) < 100: results[col] = {'psi': None, 'status': 'INSUFFICIENT_DATA'} continue
# Compute PSI using training distribution as reference breakpoints = np.percentile(train_vals, np.linspace(0, 100, 11)) breakpoints[0], breakpoints[-1] = -np.inf, np.inf
exp = np.clip(np.histogram(train_vals, bins=breakpoints)[0] / len(train_vals), 1e-6, None) act = np.clip(np.histogram(serve_vals, bins=breakpoints)[0] / len(serve_vals), 1e-6, None) psi = float(np.sum((act - exp) * np.log(act / exp)))
status = 'STABLE' if psi < 0.1 else 'INVESTIGATE' if psi < 0.2 else 'INCIDENT' results[col] = {'psi': round(psi, 4), 'status': status}
return results ```
Prevention through feature stores:
A proper feature store provides a single Python function that:
The code path is identical. Bugs are caught once. Deployment is safer. The engineering cost is front-loaded (weeks of setup). The cost of not having one compounds indefinitely (months of debugging).
The production checkpoint:
Before you ship a model: 1. Log the serving feature vector for 100 predictions 2. Recompute those features using the training pipeline on the same data 3. Compare: if any feature differs by > 5% in normalised space, fix the serving code 4. Ship feature logging infrastructure alongside the model 5. Set up daily PSI monitoring on the top-10 features by importance