Feature Engineering in Production

Online vs offline features, point-in-time joins, backfill risk

A fraud model ships with 94% offline AUC. In production it scores 82%. No code changed. No error fired. The model is even getting the exact feature it trained on — "number of transactions in the last 7 days" — but the *value* is wrong.

At training time that number came from a SQL query over a history table: an exact count. In production it comes from a Redis structure that counts approximately to stay fast. For most users the two agree. For high-volume accounts — the very users most likely to be fraud — the approximation drifts furthest from the true count, since probabilistic counters like HyperLogLog trade precision for speed exactly where volume is highest. The model was taught to trust these counts as exact, and now it is scoring live traffic on values it never learned from. Twelve points of accuracy, gone. Not a model failure — an infrastructure failure.

The same five culprits, in feature terms

*Language:* the Python training code and the Java serving code implement one formula two ways, and the small differences add up. *Data source:* batch SQL and real-time Redis handle nulls and completeness differently. *Timestamp:* training computes the feature at label time, serving computes it at request time hours later — for a rolling 7-day window that gap matters. *Preprocessing:* a scaler saved under one library version can load differently under a newer one, shifting every value silently. *Leakage:* a training feature used data that won't exist at serving time, inflating the offline score.

One shared computation path removes four of the five at once — language, data source, timestamp, and preprocessing all collapse into a single well-tested implementation: a single library or service computes "transactions in last 7 days" identically whether the caller is the training pipeline or the live endpoint. Leakage needs one more discipline on top of that: even a single shared function will leak if it's called with a wrong or late as-of timestamp, so point-in-time join enforcement is required in addition to the shared path, not replaced by it.

Point-in-time joins: the rule that keeps training honest

Here is the subtle one. When you build a training row for a fraud event at time T, you must join only feature values that existed *before* T. Grab a value from T + 1 hour — easy to do by accident, because the training job ran later — and you've leaked the future into the past. For a rolling 7-day window computed an hour late, the count can include transactions that happened *after* the event you're trying to predict. The model learns a pattern built partly on future data. In production it never has the future, so the offline number is a mirage and production comes in lower.

One warning worth internalizing. "It's the same logic, just in another language" is a hope, not a fact — it's only true once you've tested that both produce byte-identical output on identical input. Re-implementing SQL window logic in Java quietly changes null handling, overflow, and rounding, and on high-value accounts those small differences can outweigh the model's entire learned signal for that user.

Key points

• Build one feature computation path shared by training and serving, even if it means running Python at serving time or invoking a service from the training pipeline. The cost of maintaining two codepaths is not the engineering time to write them — it is the silent accuracy degradation that accumulates once they diverge. Every language boundary, data source switch, and library version difference is a potential skew source. One path eliminates the language, data source, timestamp, and preprocessing sources of skew structurally — leakage still needs point-in-time join enforcement on top of it (see the trap below).
• Trap: point-in-time join mistakes are a common, easy-to-miss leakage source in production ML. When constructing a training row for an event at time T, join only features that were computed and available before T. Using a feature from after T — even by one second — is leakage. The model learns a relationship that includes future information. In production it never has that information. Offline AUC looks great; production performance is worse than expected, and you spend weeks debugging the wrong things.
• Diagnostic: run both pipelines on the same 1,000 held-out examples and compute mean absolute difference per feature. Any feature with MAD greater than 1% of its standard deviation is a training-serving skew candidate. Sort by feature importance. A high-skew feature that ranks 47th in importance is noise; a high-skew feature that ranks 2nd is your accuracy gap. This narrows the investigation from "something is wrong" to "it is this specific feature, and here is the computation difference that causes it."

Recap

• Same feature, wrong value: SQL exact count at train vs Redis approximate count at serve → 94% AUC drops to 82%.
• Worst on high-volume accounts — the very users most likely to be fraud.
• Five culprits, feature terms: language, data source, timestamp, preprocessing, leakage. One shared path kills four; leakage still needs point-in-time joins.
• Point-in-time join = the honesty rule: for an event at T, join only values that existed *before* T.
• Rolling window leak: feature computed an hour late can include post-event transactions → offline AUC is a mirage.
• Diagnostic: run both pipelines on same 1,000 examples, MAD per feature; MAD > 1% of std = skew candidate, sort by importance.
• "Same logic, other language" is a hope, not a fact — only true once byte-identical output is tested.

Check your understanding