Real-Time Feature Engineering: Latency, Correctness, and the Streaming Trap
You can compute the feature. The question is whether you can compute it at 10ms p99 with correct training-serving parity. Most streaming feature pipelines fail at the second requirement. Here is the latency budget, the point-in-time correctness problem, and when streaming is actually worth it.
The gap between "I can compute this feature" and "I can compute it at 10ms p99 with correct training-serving parity" is where most ML systems break in production. This post covers that gap.
The latency budget
A real-time scoring call has a budget. For fraud: 30ms. For ad serving: 10ms. For search reranking: 50ms. The feature engineering layer must fit inside this budget alongside model inference, network I/O, and serialisation.
Rule of thumb: features get 30–40% of the latency budget. At 10ms, that's 3–4ms for all feature retrieval and assembly. This rules out: SQL joins, cross-service API calls, anything that recomputes from raw events at inference time.
What works within budget: Redis point reads (1–2ms), in-process computation from a payload already in memory, pre-scored embeddings looked up by key.
What needs to be real-time
Not all features need to be real-time. The question is: does staleness materially hurt model quality or create business risk?
Must be real-time: fraud velocity signals (count of transactions in the last 10 minutes — stale signals let fraud through), session context (last 3 clicks in this session — stale context makes recommendations irrelevant), real-time inventory (showing an out-of-stock item because the feature is 2 hours stale).
Can be batch: user lifetime value, 30-day engagement history, demographics, item embeddings (item inventory rarely changes meaning within hours), global statistics (average order value by category).
The mistake: building a full streaming pipeline for every feature because streaming feels more "real-time." Streaming infrastructure (Kafka + Flink + state management) is expensive, operationally complex, and introduces new failure modes. Default to batch. Move to streaming only when you can show that staleness is costing you.
The streaming architecture
Events → Kafka → Flink (windowed aggregations) → Redis (precomputed feature storage) → Serving layer (feature lookup at inference time).
Flink computes windowed aggregations: "count of transactions per user in a sliding 10-minute window." The result is a keyed state: {user_id → count}. On each new event, Flink updates this state and writes the result to Redis. The serving layer reads from Redis, not from Flink directly.
This decoupling matters: Flink is a stateful computation engine, not a low-latency lookup store. Never query Flink directly at serving time — it's designed for throughput, not p99 latency.
Point-in-time correctness
This is the hardest correctness problem in streaming features. When you build a training dataset, every feature value must reflect what the model would have seen at the time of the label event — not what was available afterward.
The failure mode: you compute "count of logins in the last 7 days" for a training example at timestamp T. But your batch job runs at T+2 hours and includes events that arrived late (timestamps < T but processed after T due to network delays). Your training feature includes 5 logins; at serving time, the model sees 3 logins (late events haven't arrived yet). The training distribution and serving distribution differ. This is point-in-time contamination.
The fix: log feature values at the time they were served to the model. Store them in a feature store with the serving timestamp. Use these logged feature values for training, not recomputed values. This is the "log-and-join" pattern used at Uber, Airbnb, and DoorDash.
Late arrivals and watermarks
Streaming engines handle out-of-order events via watermarks: a signal that says "we've received all events with timestamp before W." Events arriving after the watermark for their timestamp are "late" and can be dropped or handled separately.
If your watermark is 2 minutes: events arriving more than 2 minutes late are dropped. If your training data was generated with a 5-minute watermark (including more late events), your training features are systematically richer than your serving features.
Production rule: use identical watermark configuration in training and serving pipelines. Prefer 1-minute watermarks for fraud, 5-minute for recommendations. Document the watermark as part of the feature contract.
Training-serving skew in streaming
The four most common skew sources in streaming feature pipelines:
1. Window boundary semantics: a "7-day rolling window" computed in batch (last 7 × 86400 seconds from query time) vs in streaming (event-time window that closes at midnight) produces different counts. At boundary events (midnight, week boundaries), the counts diverge by up to 5%.
2. Null/missing handling: production Kafka messages have missing fields 0.1–1% of the time (schema drift, partial failures). Training Parquet files were cleaned — null values were imputed or dropped. The model has never seen the null representation that serving uses.
3. Encoding divergence: training uses sklearn's LabelEncoder with a fixed vocabulary. A new category appears in production. The serving encoder maps it to 0 (OOV bucket). The model interprets 0 as the first category, not as unknown. Silent wrong encoding.
4. Feature version mismatch: feature v1 in training, feature v2 in serving (after a schema change). If the feature store doesn't version features, served values silently differ from training values.
The feature store as the single source of truth
A feature store solves the training-serving skew problem by providing a single computation and storage layer. Training reads from the feature store's point-in-time API (give me feature values as of timestamp T). Serving reads from the feature store's online API (give me current feature values for entity E).
The feature store's job: maintain both the offline (historical) and online (low-latency) representations of every feature, with identical computation logic.
Feast, Tecton, and Vertex Feature Store all implement this. The critical capability: the offline store (for training) and online store (for serving) run the same feature transformation code. "Same code, two paths" is the invariant that prevents skew.
Try on Colab: build a minimal feature parity test. Create a batch pipeline that computes "7-day rolling click count per user" from a Parquet file. Create a streaming simulation that processes the same events in order with a 1-minute watermark. For 100 users, compare the feature values from both pipelines. Find at least one user where the counts differ by more than 0 and explain which boundary condition caused it.