Sequential & Session-Based RecSys

GRU4Rec, SASRec, next-item prediction, short vs long-term intent

Two-tower and matrix-factorization models treat a user as a static bag of past items — they know *what* you clicked but throw away the *order*. That's wrong for intent. A shopper who viewed [tent, sleeping bag, hiking boots] in that order is mid-mission; the same three items reshuffled tells a different story, and the *next* item (a headlamp) is predictable only from the sequence.

Session-based models predict the next item from the ordered history. GRU4Rec runs a recurrent network over the session, carrying a hidden state that summarizes everything seen so far; SASRec (and transformers4rec) replaces recurrence with self-attention so each position can look back at any earlier item directly. Self-attention wins at scale because it captures long-range dependencies without the vanishing-gradient decay a GRU suffers over a 50-event session.

Short-term vs long-term intent are two different signals that must be fused. Your *long-term* profile says you love indie films; your *current session* is 4 straight cooking videos — right now you want a fifth cooking video, not an indie trailer. A pure long-term model ignores the session; a pure session model forgets you the moment you leave. Production systems concatenate a long-term user embedding with a session-encoded state, letting the ranker weigh "who you are" against "what you're doing now."

Worked scale: a session of length L=50 in a d=128 model costs O(L²·d) ≈ 50²·128 ≈ 320k multiply-adds for one self-attention layer — trivial per request. That cheapness is why SASRec-style models moved from retrieval-only into ranking features: the sequence encoder runs in a few hundred microseconds and its output is just another embedding the funnel already knows how to consume.

Key points

• Order carries intent that a bag-of-items loses. [tent → sleeping bag → boots] implies a camping trip and a predictable next item; the same set unordered does not. Sequential models keep the order; MF/two-tower discard it. Use sequential features when the *next action* depends on the *recent trajectory*, not just the lifetime aggregate.
• GRU4Rec (recurrence) vs SASRec (self-attention): attention wins on long sessions. A GRU compresses history into one hidden state and decays old items via vanishing gradients; self-attention lets position t attend directly to position 1, so a 50-event session keeps early signal. transformers4rec is the productionized transformer variant.
• Fuse short-term (session) and long-term (profile) — neither alone is enough. The session captures *what you're doing now*; the long-term embedding captures *who you are*. Concatenate both into the ranker so a 4-video cooking binge can override a lifetime indie-film preference for the next slot, without erasing the profile.
• Sequence encoders are cheap enough to sit in ranking, not just retrieval. O(L²·d) for L=50, d=128 is ~320k FLOPs — sub-millisecond — so the encoded session becomes just another feature the funnel consumes.

Recap

• Order carries intent that a bag-of-items loses: two-tower and matrix-factorization treat a user as a static bag — they know *what* you clicked but throw away order. [tent → sleeping bag → boots] is mid-mission and predicts a headlamp; the same three reshuffled tells a different story. Use sequential features when the *next action* depends on the *recent trajectory*.
• Session-based models predict the next item from the ordered history: GRU4Rec runs a recurrent net carrying a hidden state that summarizes everything seen; SASRec / transformers4rec replaces recurrence with self-attention so each position can look back at any earlier item directly.
• Self-attention beats GRU on long sessions: a GRU compresses the whole history into one recurrently-updated hidden state, so early items decay through vanishing gradients; self-attention lets position t attend directly to position 1, keeping early signal alive over a 50-event session.
• Fuse short-term (session) and long-term (profile) — neither alone is enough: your long-term profile says "indie films," but 4 straight cooking videos means you want a fifth *now*. Concatenate both into the ranker so the in-session state can override lifetime taste for the next slot, without erasing the profile that survives for tomorrow.
• Sequence encoders are cheap enough for ranking, not just retrieval: one self-attention layer at L=50, d=128 costs O(L²·d) ≈ 320k multiply-adds — sub-millisecond — so the encoded session becomes just another embedding the funnel already knows how to consume.

Check your understanding