Real-Time ML
Streaming features, latency SLAs, caching, async fan-out, fallbacks
A fraud model that answers in 500ms is useless — the transaction clears in 200ms or the user gives up. A recommender that takes 2 seconds has already lost the session. This is the one hard constraint batch ML never faces: the prediction must arrive *before the user's patience runs out.*
Everything that makes the model better makes it slower. Every extra feature buys accuracy and costs latency; every extra layer, the same. So real-time design isn't vague tradeoffs — it's *numbers*: not "this adds latency" but "this adds 15ms, the SLA is 50ms, so 35ms is left for everything else." You budget milliseconds like money, because you can't spend what you don't have.
Async fan-out is non-negotiable. Issue all feature-store requests in parallel and wait for the *max*, not the *sum*: four 8ms features in parallel cost 8ms, in serial 32ms. Pair each with a timeout that returns a default rather than blocking past the SLA. Precompute what you can — stream user aggregates into a cache so serving is a lookup plus a light adjustment, not a heavy recompute.
Define the fallback before the incident, not during it. Every real-time system needs a documented answer to "what happens when the model endpoint is slow or down": popularity response, rule-based score, or last cached prediction, behind a circuit breaker that trips to the fallback when error rate crosses a threshold. The organizational catch: model-builders optimize accuracy, serving-builders optimize latency, and unless the budget is explicit and shared, each optimizes its own half and the system misses the SLA that only exists when you add both.
Key points
- The latency budget is a hard per-component allocation set before deployment. A 50ms fraud budget: features 5–10ms, inference 10–20ms, network 2–5ms, serialization 1–2ms. If inference alone eats 40ms, there's no room for features. Profile end-to-end in production, measured at P99 (the 99th-percentile latency — the slow-tail request that determines whether the SLA is actually met, not the average): a typical fraud-scoring trace splits like 27ms of feature-store round-trips against 9ms of inference out of a 40ms total, which is why the bottleneck is usually feature retrieval, not inference.
- Async fan-out turns a sum into a max. Four 8ms features cost 8ms in parallel, 32ms in serial. Every multi-source system must fan out and time out (slow source → default value, not a blocked request).
- A predefined fallback + circuit breaker is mandatory. Without it, engineers improvise under incident pressure and make it worse. Define the degraded response (popularity/rule/cache) and trip to it automatically when error rate spikes.
Real-time ML is latency budgeting in milliseconds: accuracy and latency trade off continuously, async fan-out converts feature-fetch cost from a sum to a max, and a predefined fallback behind a circuit breaker is what keeps the system answering when the model can't — with the budget made explicit so accuracy and serving teams don't each optimize half.
Recap
- Real-time ML has one hard constraint batch never faces: the prediction must arrive before the user's patience runs out. A fraud model answering in 500ms is useless when the transaction clears in 200ms; a 2-second recommender has already lost the session.
- Budget milliseconds like money, not vibes: everything that makes the model better (more features, more layers) makes it slower, so design in numbers — "this adds 15ms, the SLA is 50ms, so 35ms is left." A 50ms fraud budget: features 5–10ms, inference 10–20ms, network 2–5ms, serialization 1–2ms. Profile end-to-end in production at P99 (the 99th-percentile latency, not the average) — a typical trace splits like 27ms of feature-store round-trips against 9ms of inference out of a 40ms total, which is why the bottleneck is usually feature retrieval, not inference.
- Async fan-out turns a sum into a max: issue all feature-store requests in parallel and wait for the *max*, not the *sum* — four 8ms features cost 8ms in parallel, 32ms in serial. Pair each with a timeout that returns a default rather than blocking past the SLA.
- Precompute and cache what you can: stream user aggregates into a cache so serving is a lookup plus a light adjustment, not a heavy recompute.
- Fallback + circuit breaker are design, not ops: every real-time system needs a documented answer to "what happens when the model endpoint is slow or down" — a popularity response, rule-based score, or last cached prediction, behind a circuit breaker that trips automatically when error rate crosses a threshold. Define it before the incident, and make the budget explicit so accuracy and serving teams don't each optimize only half.
Check your understanding
Q1. Select the two correct parts of the fix when a fraud model has 150ms P99 against a 50ms SLA.
- A) Profile to attribute the 150ms across features, inference, and network before changing anything else.
- B) Cache precomputed feature aggregates so serving is a lookup, and fetch the remaining live features with async parallel calls plus per-source timeouts instead of serial round-trips.
- C) Move fraud scoring entirely to a nightly batch job and use yesterday's risk score at transaction time.
- D) Add a request queue and process fraud checks strictly sequentially to remove concurrency spikes.
Q2. Four feature sources each take 8ms. Your service fetches them one after another and blames the model for a 40ms+ latency. What's the actual problem?
- A) The model itself is genuinely too slow; the correct fix is quantizing its weights down to INT8 precision.
- B) Serial fetch pays the sum (4×8=32ms) instead of the max; async fan-out with per-request timeouts drops it to ~8ms.
- C) 8ms per feature lookup is simply impossible at this scale; the feature store must be badly misconfigured somewhere.
- D) Reduce down to a single feature source entirely to cut latency, accepting whatever accuracy loss results.
Q3. Why is a documented fallback + circuit breaker considered part of the *design*, not an ops afterthought?
- A) It's genuinely an ops-only concern; the design is considered done once the model meets the SLA in load tests.
- B) The availability contract depends on it: the degraded-response choice determines what users see, and a circuit breaker automates the switch.
- C) Because a documented fallback improves the model's offline accuracy metrics reported during the full evaluation and benchmarking phase cycle.
- D) Because regulators require a fallback flowchart to be included in the written design document before launch.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →