Foundational 30 min read RecSyscandidate generationrankinglatency budget

The Two-Stage Architecture

Why candidate generation → ranking exists — latency forces a cheap recall stage before an expensive precision stage

Every serious recommender answers one question: "of the millions of things I could show this user, which handful do I show *now*?" The naive answer — score every item with your best model and take the top-k — is arithmetically impossible, and understanding *why* is the whole foundation of RecSys design.

The latency wall is the forcing function. Suppose a good ranker takes 1ms to score one (user, item) pair — generous, since real cross-feature rankers are slower. A catalog of 10M items would need 10,000,000 ms = 10,000 seconds of compute per request. Your latency budget is ~100ms end-to-end. You are off by five orders of magnitude. No amount of hardware closes a 100,000× gap at request time; you cannot buy your way out of an asymptotic mismatch.

So the system splits into two stages with opposite objectives. *Candidate generation* (retrieval) cheaply narrows 10M → a few hundred using methods so cheap they can touch every item — embedding lookups, approximate nearest neighbor, precomputed lists. It optimizes recall: don't lose the good items. *Ranking* then runs an expensive, feature-rich model over only those few hundred survivors and optimizes precision: order them exactly right. The cheap stage runs over everything; the expensive stage runs only over what the cheap stage kept.

The consequence that trips up juniors: the two stages have different metrics because they have different jobs. Retrieval is judged on recall@k (did the relevant items make the shortlist?), ranking on NDCG/precision (are they ordered well?). And retrieval's recall is a *ceiling* — an item retrieval drops is gone forever; no ranker can order an item it never received. A brilliant ranker on top of a mediocre retriever is capped by the retriever. This is why "great model, mediocre recommendations" almost always means: audit retrieval recall first.

That answers *which stage broke* — it doesn't answer *how many candidates stage 1 should even keep*. Keep too few and precision-optimized ranking never sees the items that mattered; keep too many and the ranking stage — the expensive one — blows its own slice of the 100ms budget. The split point is a real design decision, and the same arithmetic that motivated the two stages in the first place answers it.

Say the team picks 500 candidates to pass forward — the number this module's own check questions already assume. The ranking stage gets roughly 20ms of the 100ms total (retrieval takes ~2ms; the rest goes to serialization, logging, and network). 20ms ÷ 500 candidates = 0.04ms per candidate — about 25× cheaper, per item, than the naive 1ms-per-pair full ranker from the latency wall above. That's the actual constraint on the ranking model: it has to be far cheaper than the heaviest model you could build, because it still runs 500 times inside one request. This is why production rankers are feature-rich but *bounded* — a few hundred features, not deep cross-attention over raw text — the per-item cost has to clear 0.04ms, not 1ms.

Now the second half of the split-point decision: what does retrieval's recall actually cost you downstream? Assume, for a given request, the truly relevant items a perfect system would show in the top-10 number 10. If retrieval's recall@500 is 0.90, only 10 × 0.90 = 9 of those 10 ideal items survive into the 500 candidates the ranker ever sees — the ranker, however good, can place at most 9 of the true top-10 correctly, a 90% ceiling on top-10 precision that no ranking sophistication can lift. Push recall@500 to 0.98, and 10 × 0.98 = 9.8 ≈ 10 of the ideal items survive — the ceiling moves to essentially 98-100%. The gap between a "good" retriever (0.90) and a "great" one (0.98) is worth roughly 8-9 points of achievable top-10 precision, bought entirely by retrieval and unrecoverable by ranking — which is exactly the earlier claim, "retrieval recall is a ceiling," made numeric.

Key points

The two-stage split is forced by arithmetic, not taste. 10M items × ~1ms/item = 10,000s per request against a ~100ms budget — a 100,000× gap. You cannot run the precise model over the full catalog, so a cheap recall stage must run first and an expensive precision stage second.
Retrieval optimizes recall; ranking optimizes precision — different jobs, different metrics. Retrieval's only sin is dropping a good item (unrecoverable); ranking's job is ordering the survivors. Judge retrieval on recall@k, ranking on NDCG/precision@k. Conflating the two is a classic interview tell.
Retrieval recall is a hard ceiling on final quality. If recall@500 = 0.7, then 30% of items the user would have loved never reach the ranker, and no ranking sophistication recovers them. Diagnose a "good ranker, bad results" system by measuring retrieval recall before touching the ranker.
The stage counts are a budget allocation. A typical split: retrieval 10M→a few hundred in ~2ms, ranking a few hundred→~100 in ~20ms. Each stage gets a hard millisecond allocation; one overrunning stage steals from the next.
The split point is a latency-budget decision, not a guess. At 500 candidates and a ~20ms ranking-stage budget, the per-item cost ceiling is 20ms ÷ 500 = 0.04ms — roughly 25× cheaper than the naive 1ms-per-pair full ranker from the latency wall. That's why production rankers use bounded feature sets, not the heaviest model imaginable.
Retrieval recall sets a numeric ceiling on top-10 precision. If the true top-10 has 10 relevant items, recall@500 = 0.90 means only 9 survive into the ranker's candidate set (10 × 0.90 = 9) — a 90% ceiling no ranking sophistication can lift. Push recall@500 to 0.98 (9.8 ≈ 10 survive) and the ceiling moves to ~98-100%: an 8-9 point swing bought entirely by retrieval, not ranking.

Takeaway

A recommender is a recall-then-precision funnel forced by a ~100,000× latency gap: cheap candidate generation maximizes recall over millions (setting an unraiseable ceiling on final quality), then an expensive ranker maximizes precision over the few hundred survivors. The split point itself is arithmetic, not guesswork: candidate count sets the ranking stage's per-item cost ceiling, and retrieval's recall sets the top-10 precision ceiling — one model cannot occupy both ends of the funnel, and no ranking improvement buys back recall retrieval already lost.

Recap

The two-stage split is forced by a latency wall, not preference: 10M items × ~1ms each ≈ 10,000s per request vs a ~100ms budget = a 100,000× gap. You cannot score the full catalog with the precise model at request time.
Two stages, opposite objectives: candidate generation (retrieval) is cheap and maximizes *recall* over millions; ranking is expensive and maximizes *precision* over the few hundred survivors. The cheap stage runs over everything; the expensive stage only over what the cheap stage kept.
Retrieval recall is an unraiseable ceiling: an item retrieval drops is gone — no ranker can order an item it never received. recall@500 = 0.7 means 30% of loved items are lost before ranking. Tell: "great ranker, mediocre feed" → audit retrieval recall first.
Different jobs → different metrics: judge retrieval on recall@k, ranking on NDCG/precision@k. Conflating them is a junior tell.
Stages are a budget allocation: e.g. retrieval 10M→a few hundred (~2ms) → rank a few hundred→100 (~20ms). Each stage has a hard ms allocation; one overrun steals from the next.

Check your understanding

Q1. An interviewer asks you to "design YouTube recommendations." You have a strong ranking model. What is the correct *first* architectural move?

A) Nightly-cache the ranker's output over the full 10M-item catalog per user via a batch Spark job, so serving becomes a single Redis lookup and session freshness is sacrificed for latency.
B) Split into a cheap candidate-generation stage (recall) feeding an expensive ranking stage (precision) — the latency budget forces the split.
C) Start with the loss function and negative-sampling scheme — e.g. BPR with 50 hard negatives per positive and temperature 0.07 — since the ranker's objective decides final quality first.
D) Pick the embedding dimension (64–128) and ANN index type (HNSW vs IVF-PQ) first, since retrieval's nearest-neighbor tuning is where nearly all engineering difficulty lives.

Q2. Your ranker scores 0.94 AUC offline, but users say the feed misses obvious interests. Retrieval recall@500 is measured at 0.6. Where is the bug?

A) The ranker — 0.94 AUC on a biased offline set is misleading; retrain with harder negatives sampled at a 4:1 ratio and recompute AUC against a held-out propensity-weighted set.
B) Retrieval — recall@500 = 0.6 means 40% of relevant items never reach the ranker, capping quality regardless of AUC.
C) The metric — AUC is the wrong choice for feeds entirely; switch to a calibrated Brier score computed over the top-500 candidates and the gap disappears without touching the system.
D) Serving — the ranker is timing out on ~40% of requests and silently returning a popularity fallback; add retries and a 50ms circuit breaker.

Q3. Select the *two* legitimate reasons candidate generation can't simply be "the full ranking model over a random 1% sample of the catalog."

A) Uniform 1% sampling has ~1% recall of the relevant items by construction — it discards ~99% of what the user actually wants before ranking ever sees it, no matter how precise the ranker is.
B) Running the full ranker (~1ms/item) over even a 1% sample (100k items of a 10M catalog) still costs ~100 seconds per request, three orders of magnitude over the ~100ms budget.
C) Random sampling corrupts the ANN index's underlying distance metric (cosine/dot-product), producing incorrect nearest-neighbor rankings even before the ranker runs.
D) It works, but only if the 1% sample is stratified by item popularity using a nightly Spark job — a cheap fix that restores full recall.

Q4. Team A improves retrieval recall@500 from 0.90 to 0.98 and changes nothing about the ranker. Team B leaves recall@500 at 0.90 and ships a much better ranking model. Assuming a true top-10 of 10 relevant items per request, which team's change can actually raise achievable top-10 precision, and by roughly how much?

A) Team A — 10 × 0.90 = 9 vs 10 × 0.98 ≈ 10 ideal items now reach the ranker, raising the achievable ceiling by roughly 8-9 points; Team B's ranker still can't rank items retrieval never delivered.
B) Team B — a better ranking model always increases NDCG regardless of what retrieval delivers, since ranking quality and retrieval recall are independent, orthogonal metrics.
C) Both equally — recall and ranking quality are two ways of expressing the same underlying precision number, so either improvement moves top-10 precision by the same amount.
D) Neither — top-10 precision is capped by the catalog size (10M), not by recall@500 or ranking quality, so neither team's change matters at request time.

Try it interactively

ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.

Open ML Systems Lab →