RecSys Stack Deep-Dive (4-Stage Funnel)

Retrieval → pre-ranking → ranking → re-ranking, latency budgets, feedback loops

The gap between a recommender *prototype* and a recommender *system* is the entire engineering stack around the model. A prototype runs a ranker over a few thousand items and prints results. Production has to serve millions of users inside a ~100ms end-to-end budget — of which the four funnel stages below account for roughly 30ms of actual compute, the remainder consumed by network round-trips between services, feature-store lookups, and serialization at each hop — A/B-test each stage independently, degrade gracefully when any component dies, and correct position bias so the ranker doesn't just resurface whatever the last model showed.

Modern stacks have four stages, not three. Between cheap retrieval and the expensive ranker sits a pre-ranking (a.k.a. coarse-ranking) stage: a lightweight model that trims thousands of retrieved candidates to a few hundred before the heavy ranker runs. Without it, the full ranker either blows the latency budget or is forced to score too few candidates. Retrieval (10M→5k, ~1ms) → pre-rank (5k→500, ~5ms) → rank (500→50, ~20ms) → re-rank (50→10, ~5ms). Re-ranking is a distinct final pass, not a smaller repeat of ranking: it takes the ranker's top ~50 scored candidates and applies constraints a per-item relevance score can't express on its own — deduplicating near-identical items, enforcing diversity across categories or sources, and injecting business rules (promotions, freshness floors, do-not-show lists) — before the top ~10 go to the user. See *reranking_diversity* for the algorithms and *recsys_feedback_loops* for how re-ranking's choices feed the position-bias problem below.

Latency is allocated, not hoped for. Each stage has a hard millisecond budget and a single overrunning stage cascades. The Rank stage's ~20ms budget, for example, splits roughly 10ms for feature retrieval and 10ms for the model's forward pass; if feature retrieval slips from 10ms to 15ms, only 5ms remains for scoring — forcing fewer candidates or a simpler model, both of which cost quality. Measure the budget end-to-end in production, never from component microbenchmarks.

The hardest correctness problem is the feedback loop. The ranker itself is a learning-to-rank (LTR) model — trained to order candidates against each other, not just score each one in isolation — and its quality is measured with rank-sensitive metrics like precision@1 (whether the single top-ranked item is actually relevant). The ranker trains on interactions shaped by what the *previous* ranker chose to show. Position 1 gets clicks regardless of quality; train on raw clicks and you teach the model to reproduce position effects, not relevance. Inverse-propensity weighting — reweighting each observed click by 1/P(click|position) to cancel out position's effect on the raw signal — and counterfactual learning are the tools that recover an unbiased relevance estimate.

Key points

• Pre-ranking is the stage most people forget. Retrieval returns thousands; the full ranker can't afford to score thousands in-budget. A cheap pre-ranker (small two-tower or GBM) cuts 5k→500 so the expensive ranker only scores hundreds. Consistency matters: if the pre-ranker and ranker disagree wildly, good candidates get cut before the ranker ever sees them (pre-ranking/ranking consistency is its own tuning problem).
• Each stage is independently trained, monitored, and deployed — that's an organizational choice as much as technical. It lets a 10-person team improve retrieval this week without re-testing ranking. Blur the boundaries and the pipeline becomes one un-shippable unit.
• Latency budgets are hard allocations measured end-to-end. A stage that overruns steals from the next. The bottleneck is usually feature retrieval, not inference — profile the whole request path in production before committing to a model size.

Recap

• Production RecSys is a 4-stage funnel, not 3: retrieval (10M→5k, ~1ms) → pre-rank (5k→500, ~5ms) → rank (500→50, ~20ms) → re-rank (50→10, ~5ms). The gap between a *prototype* and a *system* is this entire stack — serving millions inside ~100ms, A/B-testing each stage, degrading gracefully, correcting position bias.
• Pre-ranking (coarse-ranking) is the stage interviews forget: a lightweight model (small two-tower or GBM) trims thousands → hundreds so the heavy ranker fits its budget — without it the ranker either blows latency or scores too few candidates. Its own tuning problem is *pre-rank/rank consistency*: if the two disagree wildly, good candidates get cut before the ranker ever sees them.
• Latency is allocated, not hoped for — a hard per-stage budget measured end-to-end. One overrunning stage cascades: the Rank stage's 20ms splits ~10ms feature fetch + ~10ms scoring, so if feature fetch slips 10ms→15ms, scoring loses 5ms and must drop candidates or simplify. Profile the whole request path in production; the bottleneck is usually feature retrieval, not inference.
• Each stage is independently trained, monitored, and deployed — an organizational choice as much as technical, letting a small team improve retrieval this week without re-testing ranking. Blur the boundaries and the pipeline becomes one un-shippable unit.
• The feedback loop is the hardest correctness bug: the ranker trains on interactions shaped by what the *previous* ranker showed, and position 1 gets clicks regardless of quality — train on raw clicks and you learn position effects, not relevance. Fix with inverse-propensity weighting (weight by 1/P(click|position)) plus occasional randomization for unbiased data.

Check your understanding