Advanced 26 min read RecSysdeep learningDLRMDINsequence models

Deep-Learning RecSys Architectures

Wide & Deep, DeepFM, DLRM, DIN, and sequence models (SASRec/BERT4Rec) — what each models and when it fits

Once retrieval has handed the ranker a few hundred candidates, the interesting modelling question becomes: *how do you turn a pile of categorical features — user id, item id, category, device, hour, the user's last 50 clicks — into a score?* The named architectures interviewers reach for are all answers to that one question, and each one is defined by *which interactions it can represent*.

Everything starts with embedding tables. A categorical feature like item_id with 10M values can't go into a network as a one-hot vector — that's a 10M-wide input. Instead each id indexes into an embedding table: a learned matrix of shape (num_ids × d), where a lookup returns a dense d-vector (d ≈ 16–128). This is the memory reality of deep RecSys: the tables, not the MLP, dominate the parameter count — a single 100M-id feature at d=64 is 6.4B parameters (~25GB in fp32), which is exactly why industrial systems (DLRM) shard embedding tables across many hosts while the dense compute stays small. A feature cross is the other primitive: the signal "this *user* likes this *category*" isn't in either feature alone; it lives in their conjunction, and an architecture is largely characterised by whether it learns crosses automatically or needs them hand-engineered.

Wide & Deep and DeepFM: memorisation plus generalisation. *Wide & Deep* (Google) runs two paths in parallel: a wide linear model over hand-crafted cross-product features (memorises specific "user_X installed app_Y" combinations seen in training) and a deep MLP over embeddings (generalises to unseen combinations via dense similarity). The wide side needs a human to specify which crosses matter. *DeepFM* removes that manual step: it replaces the wide part with a Factorization Machine that learns *all* pairwise (2nd-order) feature interactions automatically through shared embeddings, then adds a deep MLP for higher-order patterns — same embeddings feed both, no cross engineering.

DLRM, DIN, and sequence models. *DLRM* (Meta) is the industrial workhorse: embed every categorical, take explicit pairwise dot products between all embedding pairs (2nd-order interaction), concatenate with dense features, and pass through an MLP — its identity is the embedding-table-memory reality above. *DIN* (Alibaba) adds local activation: instead of pooling a user's behaviour history into one fixed vector, it runs attention over the history with respect to the candidate item, so a user's past interest in *running shoes* is up-weighted when scoring a *sneaker* and ignored when scoring a *blender* — the user representation becomes candidate-dependent. *Sequence models* go further and model *order*: SASRec uses unidirectional (causal) self-attention over the interaction sequence to predict the next item (left-to-right, like a language model); BERT4Rec uses bidirectional self-attention with a masked-item ("cloze") objective, seeing both past and future context during training — stronger representations, but it can't be used autoregressively for pure next-item prediction the way SASRec can. The judgement call: a plain two-tower + GBDT ranker is an excellent, cheap default; you reach for these when *feature crosses matter and you don't want to hand-engineer them* (DeepFM/DLRM), when *the recent-history-vs-candidate interaction is the dominant signal* (DIN), or when *sequential order carries the intent* (SASRec/BERT4Rec).

Key points

Embedding tables are the memory reality of deep RecSys. Each categorical id indexes a learned (num_ids × d) matrix; the tables — not the MLP — dominate parameters (100M ids × d=64 ≈ 6.4B params, ~25GB). This is why DLRM-scale systems shard embedding tables across hosts while dense compute stays small.
Wide & Deep vs DeepFM = the cost of feature crosses. Wide & Deep memorises via hand-crafted cross-product features (wide) + generalises via an embedding MLP (deep); DeepFM replaces the manual wide part with a Factorization Machine that learns *all* 2nd-order crosses automatically through shared embeddings — no cross engineering.
DLRM makes the interaction explicit; DIN makes the user representation candidate-dependent. DLRM takes pairwise dot products between all embedding pairs, then an MLP. DIN runs attention over the user's behaviour history *w.r.t. the candidate* (local activation), so relevant past behaviour is up-weighted per candidate instead of pooled into one fixed vector.
SASRec vs BERT4Rec = unidirectional vs bidirectional sequence modelling. SASRec uses causal (left-to-right) self-attention to predict the next item, so it's naturally autoregressive; BERT4Rec uses bidirectional self-attention with a masked-item (cloze) objective, seeing future context in training for stronger representations but not usable for pure autoregressive next-item generation.
A two-tower + GBDT ranker is the right default; reach for DL architectures for a specific reason. DeepFM/DLRM when un-engineered feature crosses matter; DIN when the history-vs-candidate interaction dominates; SASRec/BERT4Rec when sequential order carries intent. Adopting a heavier architecture without one of those reasons buys cost, not accuracy.

Takeaway

Deep-learning RecSys architectures are all answers to "how do you turn categorical features into a score," and each is defined by which interactions it represents: Wide & Deep (hand-crafted crosses + embedding MLP), DeepFM (FM learns all 2nd-order crosses automatically), DLRM (explicit pairwise dot products, with embedding tables as the memory reality), DIN (attention over history w.r.t. the candidate → candidate-dependent user vector), and SASRec/BERT4Rec (unidirectional vs bidirectional self-attention over the interaction sequence). A two-tower + GBDT ranker is the cheap strong default; you upgrade only when crosses, history-vs-candidate, or order is the dominant signal.

Recap

Embedding tables are the memory reality: each categorical id indexes a learned (num_ids × d) matrix; tables dominate params (100M ids × d=64 ≈ 6.4B, ~25GB), so DLRM-scale systems shard tables across hosts while dense compute stays small. A feature cross ("user × category") lives in the conjunction, not either feature alone.
Wide & Deep vs DeepFM: Wide & Deep = hand-crafted cross features (wide, memorises) + embedding MLP (deep, generalises). DeepFM replaces the manual wide part with a Factorization Machine that learns *all* 2nd-order crosses automatically via shared embeddings — no cross engineering — plus a deep MLP.
DLRM vs DIN: DLRM embeds every categorical, takes explicit pairwise dot products (2nd-order interaction), then an MLP. DIN adds local activation — attention over the user's behaviour history *w.r.t. the candidate item* — making the user representation candidate-dependent instead of a single pooled vector.
SASRec vs BERT4Rec: both model *order* via self-attention over the interaction sequence. SASRec = unidirectional/causal (next-item, naturally autoregressive); BERT4Rec = bidirectional with a masked-item (cloze) objective (stronger representations from future context in training, not a pure autoregressive generator).
Default and upgrade rule: two-tower + GBDT is the cheap strong default. Upgrade to DeepFM/DLRM for un-engineered crosses, DIN for a dominant history-vs-candidate interaction, SASRec/BERT4Rec when sequential order carries intent — otherwise you buy cost, not accuracy.

Check your understanding

Q1. In a deep RecSys ranker with a few dense features and several high-cardinality categorical features (user_id ~100M, item_id ~10M), where does almost all the parameter count and memory live, and what is the standard consequence?

A) In the MLP layers — a typical 4-layer, 1024-wide tower is parameter-heavy, so the standard fix is magnitude-based pruning of the smallest-weight hidden units after training.
B) In the embedding tables — a 100M-id feature at d=64 is ~6.4B parameters (~25GB fp32); DLRM-style systems shard tables across hosts while dense compute stays small.
C) In the self-attention layers, whose compute and memory scale quadratically with the number of input features, dominating even a modest 20-feature ranker.
D) In the output softmax layer over the full 10M-item catalog, which is exactly why a hierarchical softmax with a binary tree over items is required for training to converge.

Q2. What precisely does DeepFM give you over Wide & Deep?

A) It adds a second, independent wide linear path on top of Wide & Deep's existing one, doubling the model's capacity to memorise rare user_X-installed-app_Y combinations seen in training.
B) It replaces the manually hand-crafted wide cross features with a Factorization Machine that learns all pairwise interactions automatically through shared embeddings.
C) It removes embedding tables entirely and represents every categorical feature as a one-hot vector instead, cutting memory usage by roughly 90% relative to Wide & Deep.
D) It introduces multi-head self-attention over the user's click history, a mechanism entirely absent from the original Wide & Deep architecture.

Q3. A user's click history contains running shoes, a cookbook, and a phone case. DIN scores two candidates: a sneaker and a blender. What does DIN's local-activation attention do that a fixed pooled user vector cannot?

A) It concatenates the entire history embedding into the MLP input unchanged for every candidate, giving the sneaker and the blender access to the identical, richer 3-item representation.
B) It attends over the history *per candidate*: up-weights running-shoes for the sneaker and largely ignores it for the blender, making the user representation candidate-dependent.
C) It sorts the click history strictly by recency timestamp and truncates to only the single most recent item, discarding the cookbook and phone-case interactions entirely.
D) It applies bidirectional self-attention across the full history sequence in both directions, which is precisely the mechanism that distinguishes DIN from SASRec's causal attention.

Q4. You want a sequential recommender you can also run autoregressively to predict the *next* item given a prefix. Select the *two* statements that correctly describe SASRec and BERT4Rec here.

A) SASRec uses unidirectional (causal, left-to-right) self-attention trained to predict the next item, so it is naturally usable autoregressively.
B) BERT4Rec uses bidirectional self-attention with a masked-item (cloze) objective, seeing future context in training, so it is not a pure left-to-right next-item generator.
C) BERT4Rec's bidirectional attention strictly dominates SASRec on every downstream task including autoregressive next-item prediction, making it the universal default choice.
D) Neither SASRec nor BERT4Rec can perform next-item prediction; that capability requires a separately-trained two-tower retrieval model with an ANN index.

Q5. Your team runs a solid two-tower retriever plus a GBDT ranker on tabular features. When is switching the ranker to DLRM or DIN actually justified, rather than cargo-culting?

A) Always — deep architectures strictly dominate gradient-boosted trees on tabular ranking benchmarks such as the Criteo and Avazu leaderboards, so any switch is by definition an upgrade.
B) When a GBDT captures a concrete signal poorly: many feature crosses it can't cheaply represent (DLRM's automatic pairwise interactions) or a dominant history-vs-candidate interaction (DIN); otherwise the DL model mostly adds cost.
C) Whenever offline AUC falls below the 0.9 threshold, since deep architectures are the only known technique capable of pushing tabular ranking AUC past that specific bar.
D) Only once the catalog exceeds roughly 1M items, since that is documented as the point at which gradient-boosted tree training stops converging within a reasonable time budget.

Q6. Which statement about feature crosses across these architectures is correct?

A) Wide & Deep and DeepFM both require the engineer to manually specify which cross-product features matter, since neither architecture has a mechanism for learning crosses on its own.
B) A raw MLP over concatenated embeddings doesn't reliably learn low-order crosses, which is why DeepFM's FM component and DLRM's pairwise dot products add a dedicated mechanism.
C) DLRM avoids modelling feature crosses entirely and relies purely on MLP depth to approximate them, which is documented as the reason it typically requires 15+ dense layers.
D) Factorization Machines are mathematically restricted to modelling 3rd-order and higher-order feature interactions, and cannot represent a simple pairwise (2nd-order) cross.

Try it interactively

ML Systems Lab is a free interview-prep platform for ML engineers — work through the full interactive module, quizzes, and drills.

Open ML Systems Lab →