The 6-Step ML System Design Framework

Clarify → scope → data → model → serving → monitoring

The most common failure in ML system design — in interviews and in production alike — isn't the model, it's a problem that was never properly scoped. An engineer proposes a transformer for something a gradient-boosted tree would have solved, or designs offline batch scoring for a system that turns out to need low-latency online inference. Both are the same mistake: jumping to architecture before the constraints are known.

The framework forces constraints to surface before any architectural decision

- Clarify the objective — the business metric and the north star are rarely the same thing: optimizing for click-through rate alone rewards clickbait, so state explicitly what "success" and "failure" mean, plus the QPS (queries per second), the latency SLA (service-level agreement), and the cost of a false positive versus a false negative. - Frame as an ML problem — ranking, classification, regression, or retrieval, and whether the answer needs a single model or a pipeline of models. - Define labels — where they come from, how fresh, how biased, and whether they even exist yet. - Feature design — which signals exist at serving time, whether they're point-in-time correct, and where leakage risk hides. - Model choice — read off the constraints already fixed, not picked first from preference. - Serving and monitoring — how the model reaches production safely and how its decay gets caught.

The single most common interview failure is skipping to step 4 or 5 before any of the first three are answered.

Why the ordering is load-bearing. Every downstream choice is a function of the earlier answers. A 200ms transformer is disqualified the instant the SLA turns out to be 50ms. A supervised model is disqualified the instant you learn there are no labels and none are coming. If you pick the model first, you discover these walls after weeks of work instead of in the first five minutes.

Step 2's "what type of problem is this?" looks obvious and isn't. Recommendation splits into retrieval (approximate-nearest-neighbor search over candidates) plus ranking (scoring the survivors point-wise, pair-wise, or list-wise) — two different problems chained together, not one. Fraud detection is binary classification, but with extreme class imbalance and an adversary actively adapting to whatever the model learns, which ordinary classification tuning doesn't account for. ETA prediction is regression over a right-skewed distribution with temporal features, where a model that's well-calibrated on typical trips can still be badly wrong on the long tail. Naming the task type also forces the single-model-vs-pipeline question: a recommendation system that skips straight to "build a ranking model" has silently decided there's no retrieval stage, which only works if the candidate pool is already small.

Step 3 is easy to treat as paperwork — it isn't. "Define labels" sounds procedural, but it decides whether the rest of the design is even possible. For recommendation, labels split into explicit (a 5-star rating) and implicit (a click, a dwell, a share) — implicit is noisier but available at scale, and the noise is real: a 2-second dwell is not a positive signal, so the exact dwell threshold has to be stated, not assumed — the worked example below fixes it at 10 seconds. For fraud detection, the label exists — a transaction is eventually confirmed fraudulent or not — but it arrives late: chargebacks land roughly 30 days after the transaction, which means today's training data is 30 days stale by definition, and the model is always learning last month's fraud patterns. For content moderation, there's no free label at all — it requires human annotators, an explicit agreement threshold between them, and a defined label latency (the time from a post going live to a label existing for it). A design that names a model architecture before answering these three questions hasn't actually started.

Step 4 turns "what features?" into a leakage-and-skew audit, not a brainstorm. Once the label is defined, every candidate feature gets checked against two questions before it's allowed into the model: is it available at serving time — computed at inference, not only reconstructable after the fact — and is it point-in-time correct, meaning no information from after the label's timestamp has bled into the row. A feature like "this post's engagement rate" computed from the post's lifetime totals leaks the very outcome the model is trying to predict back into training. A feature like "average session duration" that only exists as a nightly batch job isn't available on the real-time serving path the production system actually uses, so the model trains on a signal it will never see live. Both failure modes get their own full treatment elsewhere — see Data Splits and Leakage and Training-Serving Skew — but step 4's job is to catch them before a single line of model code is written.

Step 5's model choice reads off the constraints already fixed in steps 1–4 — it is not a menu picked by preference. Low latency plus mostly tabular features points to a gradient-boosted tree, a GBM (LightGBM, XGBoost): fast at inference, and interpretable enough to explain a single prediction. A retrieval stage scoring millions of candidates points to a two-tower network, trained so that a nearest-neighbor search over precomputed embeddings can substitute for scoring every item individually — the only way the latency budget survives that scale. A ranking stage re-scoring a short list of finalists points to a transformer over the user's interaction sequence, because order and recency carry signal a tabular model would discard. Small data with a hard interpretability requirement points to logistic regression over handcrafted features, even though it will never top a leaderboard. Large-scale unstructured text points to a fine-tuned or instruction-tuned LLM. Every one of these is disqualified or selected by a constraint fixed earlier — the latency SLA, the label volume, the feature space — never by which architecture is newest.

Step 6 is a staged trust process, not a single deploy. Offline evaluation runs against a time-ordered holdout — never a random split, which would leak future rows into the training set — scored on a metric chosen for the actual task (NDCG@10 — a ranking-quality score that rewards placing the right items near the top of the list, evaluated over the top 10 results — for a ranked list, not plain accuracy, which can't see ordering at all). Online, the model earns trust in stages: shadow mode first, where it scores live traffic and every prediction is logged but nothing reaches a user; then canary, where it serves a small slice, typically 5–10% of traffic; then a champion-challenger promotion, where the new model only replaces the current one after it beats it on the live business metric, not the offline one. Every stage has a defined rollback trigger, decided before launch — see Offline vs Online Evaluation for why the offline number and the online number routinely disagree, and why skipping straight from offline evaluation to full rollout is how a model that looked great on a holdout set quietly tanks a real metric.

Worked example: ranking a news feed. Step 1 — the objective is weekly active days, not raw click-through rate, because CTR alone rewards outrage bait over content people actually come back for. Step 2 — this is list-wise ranking over candidate posts, built as a retrieval-then-ranking pipeline: retrieval narrows millions of posts to a shortlist, ranking orders that shortlist precisely. Step 3 — labels are implicit and available in real time: a 10-second-plus dwell counts as a positive, a scroll-past counts as a weak negative, a share counts as a strong positive. Step 4 — the feature set is a user's historical-engagement embedding, a BERT embedding of the post text, a user-post affinity score, a recency-decay term, and a source-reliability signal; each one passes the step-4 leakage/serving-time check before it's allowed in — "this post's engagement rate" would fail it and stays out. Step 5 — a two-tower network handles retrieval, a 6-layer transformer over the user's interaction sequence ranks the top 100 survivors, and the whole pipeline retrains daily because engagement patterns genuinely shift day to day. Step 6 — NDCG@10 is the offline metric; the online A/B test measures weekly active days with a 2-week minimum exposure window, because a habit change doesn't show up in a single day's data.

Key points

• Steps 1–3 constrain every architecture decision: clarify, frame, and plan labels before touching a model. Without QPS, latency SLA, label availability, and the cost asymmetry between error types, every later decision is a guess. Concretely: a "design a spam filter" prompt has no single right answer until you know whether it blocks the email synchronously (needs <100ms) or quarantines async (can take seconds), and whether a false positive (real mail lost) costs more than a false negative (spam delivered). That asymmetry can even dictate the architecture itself — e.g. a two-stage design (cheap model auto-approves, expensive model reviews the rest) plus a human-in-the-loop — decided in step 1, not tuned later as class weights inside the model.
• Step 3's labels aren't just "available or not" — they carry a domain-specific cost that shapes the rest of the design. Recommendation labels are implicit and noisy (a 2-second dwell isn't a real positive); fraud labels are correct but structurally stale (chargebacks confirm fraud roughly 30 days after the transaction, so training data always lags current fraud patterns); content-moderation labels need human annotators with an explicit agreement threshold and a defined label latency. Naming a model before answering which of these applies means guessing at a problem you haven't actually defined yet.
• Step 4 is a leakage-and-skew audit, not a feature brainstorm. Every candidate feature has to pass two checks before it enters the model: available at serving time (computed at inference, not reconstructable only after the fact), and point-in-time correct (no information from after the label's timestamp bleeds into the row). A feature built from a post's lifetime engagement totals leaks the outcome back into training; a feature that only exists as a nightly batch job isn't there on the real-time serving path. These two failure modes are big enough to be their own modules — see Data Splits and Leakage and Training-Serving Skew.
• Steps 5 (model) and 6's serving half are coupled, not sequential: model choice and serving architecture must be solved together. A model whose memory footprint exceeds your target serving hardware's capacity can't run there as a single instance; a 200ms model cannot serve real-time. The model itself is read off the constraints already fixed — low latency + tabular features points to a GBM, million-item retrieval points to a two-tower network, a short ranking list points to a transformer over the interaction sequence, small data with interpretability needs points to logistic regression — never picked first because it's newest.
• Step 6's monitoring half is not optional, and rollout itself is staged, not a single deploy. A deployed model has no built-in signal for its own decay: without monitoring input distributions, prediction distributions, and the business metric, the first symptom of failure is a revenue drop days after the damage began. Getting to that deployed state is itself gated — shadow mode (logged, never shown to users), then canary (5–10% of traffic), then a champion-challenger promotion once the new model beats the live business metric, each stage with its own rollback trigger defined before launch.

Recap

• The 6 steps in order: clarify the objective → frame as an ML problem → define labels → feature design → model → serving/monitoring. Jumping straight to model (step 4/5) is the single most common interview failure — the classic junior tell.
• Steps 1–3 are load-bearing constraints, not preamble: QPS, latency SLA, label availability, and the FP-vs-FN cost asymmetry disqualify most architectures up front. A 200ms transformer dies the instant the SLA turns out to be 50ms; a supervised model dies the instant you learn there are no labels coming.
• Step 3's label strategy is domain-specific, not binary: recommendation → implicit/noisy (dwell threshold must be stated); fraud → correct but ~30 days stale (chargeback delay); content moderation → human-annotator-bound with an agreement threshold and defined label latency.
• Step 4 is leakage + serving-skew, checked per feature: available at serving time? point-in-time correct? A feature built from a post's own lifetime totals leaks the label; a nightly-batch-only feature isn't there for real-time serving. Full depth: Data Splits and Leakage, Training-Serving Skew.
• Step 5's model choice is read off steps 1–4, coupled to step 6's serving half: latency+tabular → GBM; million-item retrieval → two-tower; short-list ranking → transformer over the sequence; small data+interpretability → logistic regression. A model whose memory exceeds the target hardware can't run there; a 200ms model can't serve real-time.
• Step 6 is staged trust, not a single deploy, plus ongoing monitoring: offline eval on a time-ordered holdout with a task-correct metric (NDCG@10, not plain accuracy) → shadow mode (logged, unseen) → canary (5–10% traffic) → champion-challenger promotion on the live business metric, each stage with its own rollback trigger. Post-launch: monitor input/prediction distributions and the business metric, or the first symptom of decay is a revenue drop days late.

Check your understanding