Two-Tower Models: How YouTube and Spotify Do Candidate Retrieval

A recommendation system cannot score all 100 million items for every user request. Retrieval narrows the candidate set from millions to hundreds. The two-tower model is the standard architecture for this step: one tower encodes the user, one encodes the item, and the dot product of their embeddings is the retrieval score. This is what YouTube's deep retrieval network, Spotify's Discover Weekly, and Pinterest's PinSage all build on.

A production recommendation system has two stages: retrieval (find hundreds of candidates from a corpus of millions) and ranking (score each candidate precisely, using expensive features). The two-tower model is the standard retrieval architecture. Understanding it from first principles reveals why it is designed this way and where it breaks down.

The retrieval problem

At request time, you need to find the k most relevant items for a user from a corpus of N items. For YouTube with 800 million videos and 200 million daily active users, you cannot run a full neural network scorer over all 800 million videos per user per request — even at 1ms per video, this is 220 hours per user. You need an architecture where item embeddings are precomputed and user queries can be matched against them in milliseconds.

The two-tower architecture

The solution: decouple the user representation and the item representation into two separate networks (towers) that produce fixed-dimensional embeddings. User tower: f(user features) → u ∈ R^d. Item tower: g(item features) → v ∈ R^d. Retrieval score: u · v (dot product, or cosine similarity).

The key property: item embeddings can be precomputed offline and indexed. At request time, only the user tower runs (the user embedding changes with context). The retrieval problem reduces to approximate nearest-neighbour search: find the k items in the index whose embeddings are most similar to the user embedding. With FAISS or ScaNN, this runs in ~10ms even for 100M items.

Training: what makes a positive pair?

Training requires positive (user, item) pairs and negative pairs. Positives are typically engagements: watches, clicks, listens, purchases. Negatives are harder. Random negatives (any item the user did not engage with) are easy but uninformative — the model trivially separates engaged content from random content. Hard negatives — items the model currently ranks highly but the user did not engage with — provide stronger learning signal and are essential for production quality. Pinterest PinSage and Google's Dual Encoder both use hard negative mining.

In-batch negatives: a practical trick

In a batch of B (user, item) positive pairs, each item in the batch serves as a negative for all other users. For a batch size of 4096, each user has 4095 negatives with no extra computation. Sampling bias correction is necessary: popular items appear more frequently in batches and are over-represented as negatives, causing the model to push their embeddings away too aggressively. A frequency-based correction weight is applied to each negative.

Feature engineering for each tower

User tower features: user ID embedding, watch/listen history embeddings (average of recently engaged item embeddings), demographic features, contextual features (time of day, device, country). Item tower features: item ID embedding, content features (text description embedding, thumbnail embedding, category), popularity statistics, content age. The towers share no parameters — they are fully separate networks.

Serving architecture

Offline: run the item tower on all items in the corpus, store (item_id, embedding) pairs in an ANN index (FAISS, ScaNN, Weaviate). Online: given a user request, run the user tower to produce u; query the ANN index for the k nearest item embeddings; return the corresponding item IDs as retrieval candidates.

The retrieved candidates (typically k=500-2000) are passed to the ranking model, which can use features unavailable to the retrieval model (e.g., user-item interaction features, expensive content analysis) because it only scores hundreds rather than millions of items.

YouTube's DNN for candidate generation (Covington et al., 2016)

YouTube's original two-tower retrieval paper uses: user tower = DNN over user history (average of video embeddings for watched videos) + demographic features; item tower = video embedding (a lookup table). Training objective: predict the next video in a watching session from the user's history. Negatives are sampled from the video corpus. The paper reports that serving with approximate nearest-neighbour search adds only 1ms latency compared to exact search.

Try on Colab: build a minimal two-tower retrieval model on the MovieLens dataset. User tower: embed user_id + average of watched movie embeddings. Item tower: embed movie_id + genre one-hot. Train with in-batch negatives. After training, index all movie embeddings with FAISS. Query: given a user who watched [movie A, B, C], retrieve the top-20 candidates. Evaluate recall@20 (how many relevant movies are retrieved).