Two-Tower Models
Encode separately, compare cheaply — the retrieval workhorse
The most accurate way to score a user against an item is to feed them into one model together so it can weigh every interaction. The problem is arithmetic: that means scoring *every* user against *every* item at query time. For 10M items at 1000 users/second, that's 10 billion joint forward passes per second — an overnight warehouse job, not real-time retrieval.
The two-tower trick: encode separately, compare with a dot product. A user tower and an item tower map into the same embedding space; similarity is a plain dot product. Because an item's embedding no longer depends on who's asking, you compute *all* item embeddings offline, once, and index them for approximate-nearest-neighbor (ANN) search. At query time you encode just the one user and look up neighbors — ~10ms across 100M items. Retrieval quality itself is measured as recall@K: the fraction of truly relevant items that land inside the top K candidates the tower hands to the ranker. A tower with recall@100 of 60% is failing to surface 40% of the relevant items before the ranker ever sees them — no ranker can recover items retrieval never returned.
What you give up, and who picks it up. Encoding the two sides apart means the model can't capture fine user×item feature interactions — exactly what the expensive joint model was good at. That job is handed downstream to the cross-attention *ranker*, which only has to look at the few hundred candidates retrieval already narrowed. Two-tower for recall, cross-encoder for precision: the same recall-then-precision split as the whole funnel, in miniature.
Key points
- Two-tower makes real-time retrieval over 100M items possible by precomputing item embeddings offline. Item embeddings don't depend on the query user, so they're computed once and ANN-indexed; retrieval is one user-embedding computation plus a lookup. Cross-attention destroys this because item encoding would depend on the user.
- In-batch softmax with hard-negative mining is the standard recipe. Other items in the batch serve as negatives; explicitly mining high-scoring-but-unclicked items forces the model to learn fine distinctions. Random negatives are too easy — the model separates a clicked video from a random one with near-zero gradient and learns nothing subtle.
- ANN index staleness scales with catalog volatility. When item features change, the indexed embedding is stale. Fast-changing catalogs (price, inventory) need delta re-embedding of changed items; stable catalogs tolerate weekly full rebuilds. Staleness is a continuous freshness-vs-cost tradeoff, not a corner case.
Two-tower breaks the user×item coupling so item embeddings can be precomputed and ANN-indexed — buying ~10ms retrieval over 100M items at the cost of fine feature interactions, which the downstream cross-encoder ranker restores over the few hundred survivors.
Recap
- Joint (cross-attention) scoring is most accurate but impossible at retrieval scale: scoring every user against every item is O(all items)/query — 10M items × 1000 users/s = 10B joint forward passes/second, an overnight warehouse job, not real-time retrieval.
- Two-tower: encode separately, compare via u·v. Item embeddings are query-independent → precompute + ANN-index once; query time = encode user + lookup, ~10ms/100M items.
- What you give up, and who restores it: encoding the two sides apart loses fine user×item feature interactions — exactly what the joint model was good at. That job is handed downstream to the cross-attention *ranker*, which only scores the few hundred candidates retrieval already narrowed. Two-tower for recall, cross-encoder for precision.
- Training recipe = in-batch softmax + hard-negative mining: other items in the batch serve as negatives; explicitly mining high-scoring-but-unclicked items forces fine distinctions. Random negatives are too easy — separating a clicked video from a random one gives near-zero gradient and teaches nothing subtle.
- Ops: ANN staleness scales with catalog volatility. When item features change, the indexed embedding is stale. Fast-changing catalogs (price, inventory) need delta re-embedding of changed items; stable catalogs tolerate weekly full rebuilds. Staleness is a continuous freshness-vs-cost tradeoff, not a corner case.
Check your understanding
Q1. Why does a cross-attention model that jointly encodes user and item fail at retrieval scale even though it's more accurate?
- A) Cross-attention overfits badly on large catalogs, so its accuracy advantage disappears entirely above roughly 1M items.
- B) Its representation depends on the query user, so embeddings can't be precomputed — every item must be scored fresh per query, orders of magnitude over budget.
- C) Cross-attention requires specialized GPUs that physically can't be co-located with the ANN index at all, adding fatal cross-datacenter network latency to every query.
- D) It can't produce fixed-length vector embeddings, so standard ANN libraries simply reject its raw output format.
Q2. A two-tower model's recall@100 — the fraction of truly relevant items that make it into its top-100 candidates — is stuck at 60%. Which single change most directly attacks that low recall?
- A) Hard-negative mining plus richer features and a larger embedding dimension, to better separate close items in the space.
- B) Expand the ANN candidate set from 100 to 500, letting the downstream ranker filter more aggressively over more survivors.
- C) Reduce embedding dim from 256 down to 64 so the ANN index shrinks and more items become individually reachable.
- D) Switch dot-product similarity to L2 distance instead, mainly for numerical stability in high-dimensional spaces.
Q3. Your item catalog updates prices every few minutes, but the ANN index is rebuilt nightly. What's the failure mode and the right fix?
- A) No failure at all — price is purely a ranking-stage feature, so a stale retrieval index is completely harmless here.
- B) Items whose relevance changed are retrieved with yesterday's stale embedding. Fix with continuous delta re-embedding of just the changed items.
- C) The dot product silently overflows whenever prices change; simply normalizing embeddings nightly fixes the overflow.
- D) It's actually the user tower that goes stale here, not the items; retraining the user tower hourly lets the item index safely stay on its nightly schedule.
Try it interactively
ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.
Open ML Systems Lab →