Features & Freshness
Real-time vs batch features, train/serve skew, the feature-freshness trap
A recommender's features come in two speeds, and the boundary between them is where most silent production bugs live. Get the freshness model wrong and your offline metrics look great while the live system quietly recommends yesterday's world.
Batch features are computed on a schedule; real-time features are computed per event. "User's 90-day purchase count" is a batch feature — recomputed nightly, cheap, stable. "Items viewed in the last 5 minutes" is a real-time feature — it must reflect *this* session, so it's computed online from a streaming store. The two live in different systems (a warehouse vs a low-latency online store), and the recommender reads both at serving time. The design question is which signals *need* to be fresh: session intent decays in minutes, so a stale "recent views" feature is nearly useless; long-term taste is stable, so a day-old "favorite genre" is fine.
Train/serve skew is the trap: the same feature computed two different ways. Training features are usually computed in a batch pipeline (Spark/SQL over historical logs); serving features are computed in a live path (streaming/online store). If the two implementations differ — a different null default, a timezone, a window boundary, a slightly different aggregation — the model trains on one distribution and serves on another. The model's live behavior degrades in a way *no offline metric can catch*, because offline evaluation uses the training-side computation. The fix is a feature store that guarantees one definition consumed identically at train and serve time.
The feature-freshness trap is skew's time-shifted cousin: point-in-time correctness. When you build training data, each label (did the user click at time t?) must be joined to feature values *as they were at time t* — not their current values. Join "user's total purchases" as of *today* onto a click from three months ago and you've leaked the future: the model learns from information it won't have at serving time, inflating offline metrics and collapsing in production. Point-in-time-correct joins (as-of joins on event timestamps) are the non-negotiable discipline that makes offline training data match serving reality.
Key points
- Match freshness to signal decay. Session-intent features (recent views, current query) decay in minutes and must be real-time; long-term taste features (favorite genre, lifetime spend) are stable and can be batch. Making everything real-time wastes infra; making everything batch kills session responsiveness.
- Train/serve skew = one feature, two implementations, two distributions. A different null-handling, timezone, or window between the training (batch) and serving (online) computation silently shifts the model's input distribution. No offline metric catches it because offline eval uses the training-side values. A feature store enforces one definition consumed identically at both times.
- Point-in-time correctness prevents label leakage in training data. Each label must be joined to feature values *as of the event's timestamp*, not their current values. As-of joins on event time are mandatory; a naive join of current features onto historical labels leaks the future and inflates offline metrics.
- The two failure modes look identical offline and different in prod. Both skew and non-point-in-time joins produce great offline numbers and a live regression — so a large offline/online gap is the diagnostic signature to check features first.
RecSys features split into batch (stable, scheduled) and real-time (session, streamed), and the two hardest bugs both hide behind good offline metrics: train/serve skew (one feature computed two ways → distribution shift no offline metric catches) and non-point-in-time joins (current features glued onto historical labels → future leakage). A feature store with as-of joins is the discipline that closes both.
Recap
- Two feature speeds, matched to signal decay: batch (scheduled, stable — 90-day purchase count) vs real-time (per-event, streamed — last-5-minutes views). Session intent decays in minutes → must be fresh; long-term taste is stable → batch is fine. Everything-real-time is expensive over-engineering; everything-batch kills responsiveness.
- Train/serve skew = one feature, two implementations: a different null default, timezone, or window between the training (batch) and serving (online) path shifts the input distribution. No offline metric catches it (offline eval uses training-side values). A feature store enforcing one definition consumed identically is the fix.
- Point-in-time correctness stops label leakage: each label must join to feature values *as of the event timestamp*, not current values. Naive joins of current features onto historical labels leak the future, inflating offline metrics and collapsing in prod. As-of joins on event time are non-negotiable.
- Both bugs share one signature: great offline numbers + a live regression. A large offline/online gap → check feature parity and join correctness first.
Check your understanding
Q1. Your recommender scores well offline but underperforms live. You discover the "items viewed in last hour" feature is computed with Spark (rounding timestamps to the hour) in training and with a streaming store (exact rolling window) at serving. What is this, and why did offline metrics miss it?
- A) Concept drift — actual user behavior shifted between the training window and the serving window; retrain nightly on the trailing 24 hours of logs to close the gap.
- B) Train/serve skew: the same feature computed two ways; offline eval used the training-side values, so the skew was invisible until live traffic hit the streaming path.
- C) Label leakage — the hourly rounding window lets a click that happens within the same rounded hour leak into its own "viewed in last hour" feature; shrink the window to 5 minutes.
- D) Overfitting — the model memorized the specific artifact of Spark's hour-boundary rounding as a spurious signal; add L2 regularization and dropout to the input layer.
Q2. When constructing training data, you join each historical click to the user's *current* lifetime-purchase count. Offline AUC jumps to 0.98; production is far worse. What went wrong?
- A) Nothing is wrong with the data; 0.98 AUC is the model's true skill, and the production gap is purely a p99 serving-latency issue that a bigger GPU fleet would close.
- B) Non-point-in-time join: the current purchase count leaks purchases made *after* the click; use an as-of join keyed to the click's timestamp instead.
- C) The join produced duplicate rows on a many-to-many key, inflating the positive class to roughly 3x its true rate; deduplicate on (user_id, click_id) and re-evaluate.
- D) AUC is structurally the wrong metric whenever a point-in-time join is involved; switch to NDCG@10 computed on the same joined table and the 0.98-vs-production gap will close.
Q3. A teammate proposes making *every* feature real-time "to be safe." Select the *two* correct reasons this is the wrong default.
- A) Freshness should match signal decay — stable long-term-taste features (favorite genre, lifetime spend) gain nothing from being recomputed per-event instead of nightly.
- B) Blanket real-time adds streaming infrastructure cost and complexity (Kafka/Flink pipelines, low-latency online stores) for features that would produce an identical value whether computed hourly or per-event.
- C) Real-time features cannot physically be stored in an offline feature store, so marking a feature real-time permanently breaks the point-in-time (as-of) joins used to build training data.
- D) Streaming engines such as Flink or Kafka Streams cannot compute windowed aggregations, so a 90-day-lifetime feature could never be produced through a real-time path at all.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →