Learning-to-Rank Systems

Pointwise vs pairwise vs listwise, LambdaMART, position bias, distillation

A trap that catches almost everyone: training a classifier to predict relevance and sorting by its score is *not* the same as training a ranker. A classifier tuned for per-item accuracy can achieve low average error across all items and still order them wrong — because ranking is *relative*. What matters is which item beats which, not the exact number on each.

Three ways to train for order. *Pointwise* scores each item alone and misses the relative point. *Pairwise* learns "A should rank above B" — fixes pairs but treats a swap at rank 1 the same as a swap at rank 100. *Listwise* optimizes the whole list, which is what you want, but it's expensive and sensitive to label noise. For tabular ranking (web search, ads) the practical winner is LambdaMART: gradient-boosted trees whose gradients are weighted by NDCG impact, so a swap near the top gets a far bigger push than one near the bottom — a ranking-aware signal without needing NDCG to be differentiable (it isn't).

The deeper problem none of these fixes alone: position bias. Click data is contaminated by *where* items were shown. Position 1 collects clicks whether or not it deserved them, so training on raw clicks teaches the model to reproduce position effects — a self-reinforcing loop where it keeps promoting whatever the last model promoted. Breaking it needs inverse-propensity weighting: weight each example by 1/P(click|position), so position-1 examples count less and position-5 examples count more.

Key points

• LambdaMART is the tabular workhorse: GBM with gradients weighted by NDCG impact. Each item's gradient sums LambdaRank pair-gradients weighted by how much swapping the pair changes NDCG. A rank-1-vs-2 swap gets a much larger gradient than rank-98-vs-99 — NDCG-aware training without a differentiable NDCG.
• Position bias is a correctness bug that loss choice alone can't fix. Position 1 gets ~10× the clicks of position 10 regardless of relevance — because users scan top-down and rarely look past the first few results, so top slots accumulate clicks from visibility alone, not just quality; raw-click training reproduces the prior model's ranking. Inverse-propensity weighting (1/P(click|position)) plus occasional randomization recovers unbiased relevance.
• Online distillation decouples quality from serving latency. A large teacher with expensive features (cross-attention, full history) trains offline; a small student matches its rankings without those features and serves fast. The standard pattern when the most accurate model is too slow to serve directly — but the student only learned to imitate the teacher on the queries it was trained on, so as live traffic drifts from that training distribution, the student's imitation quality degrades even though it still matches the teacher on held-out data (distribution shift between training and serving).

Recap

• Sorting a classifier ≠ training a ranker: ranking is *relative* — what matters is which item beats which, not the exact score on each. A classifier tuned for per-item accuracy can achieve low average error across all items and still misorder the few hard, high-value pairs near the top.
• Three ways to train for order, cost and fidelity rising together: *pointwise* scores each item alone (misses the relative point); *pairwise* learns "A ranks above B" (fixes pairs but treats a rank-1 swap like a rank-100 swap); *listwise* optimizes the whole list (what you want, but expensive and label-noise sensitive).
• LambdaMART is the tabular workhorse: gradient-boosted trees whose gradients are weighted by |ΔNDCG| — a swap near the top gets a far bigger push than one near the bottom, injecting a ranking-aware signal without needing NDCG to be differentiable (it's sorting-based, zero-gradient almost everywhere).
• Position bias is a correctness bug loss choice alone can't fix: position 1 gets ~10× the clicks of position 10 regardless of relevance — top slots accumulate clicks from visibility alone since users scan top-down and rarely reach lower results — so raw-click training reproduces the prior model's ranking in a self-reinforcing loop. Break it with inverse-propensity weighting (weight each example by 1/P(click|position)) plus occasional randomization — not a better loss.
• Online distillation decouples quality from serving latency: a large teacher with expensive features (cross-attention, full history) trains offline; a small student matches its rankings without those features and serves fast. The standard pattern when the most accurate model is too slow — watch for train/serve distribution shift as live traffic drifts from the distillation set.

Check your understanding