Learning to Rank: What NDCG Is Measuring and How LambdaRank Optimises It

Ranking is not classification. You are not predicting whether a user will click — you are ordering a list so that the most relevant items appear at the top. The metrics that measure ranking quality (NDCG, MAP, MRR) are non-differentiable. LambdaRank is the engineering solution that makes it possible to optimise these metrics directly. This is how web search ranking, recommendation ranking, and ad ranking work at FAANG.

A ranking model takes a query (user + context) and a list of candidates and produces an ordering. The goal is to put the most relevant items at the top. This sounds like classification, but the evaluation metric depends on position — a relevant item at rank 1 is far more valuable than one at rank 10. Standard classification losses do not capture this.

NDCG: the standard ranking metric

Normalised Discounted Cumulative Gain (NDCG) measures ranking quality with position-aware discounting. For a ranked list of items with relevance labels r_1, r_2, ..., r_k:

DCG@k = Σ_{i=1}^{k} (2^{r_i} - 1) / log_2(i + 1)

Items at lower positions are discounted logarithmically. An item with relevance 3 at position 1 contributes 7/1 = 7; the same item at position 5 contributes 7/2.585 = 2.71. NDCG normalises by the ideal DCG (IDCG) — the DCG of the perfect ranking: NDCG@k = DCG@k / IDCG@k, giving a value in [0, 1].

NDCG handles graded relevance (not just binary relevant/not): a 4-star relevance item is worth more than a 3-star item. This is important for search where relevance is not binary — some documents are perfect answers, some are tangentially related.

MAP: for binary relevance

Mean Average Precision (MAP) is the mean over queries of the Average Precision per query. Average Precision: compute precision at each position where a relevant item appears, then average. MAP is appropriate when relevance is binary. It rewards finding all relevant items, not just the top ones, and rewards finding them early.

The non-differentiability problem

NDCG is non-differentiable. It depends on the rank position of each item, which is a discrete quantity that changes discontinuously as scores change. You cannot compute ∂NDCG/∂score and use gradient descent directly.

Three approaches to learning-to-rank

Pointwise: treat each (query, document) pair independently as a regression or classification problem. Predict a relevance score; rank by scores. Loss is MSE or cross-entropy. Simple but ignores the list structure — optimising individual relevance scores does not guarantee the ranking is good.

Pairwise: for each pair of documents (i, j) where i is more relevant than j, train the model so score(i) > score(j). RankNet (Burges et al., 2005) uses a pairwise cross-entropy loss. Pairwise training considers relative order but still does not account for position: it is equally costly to invert positions 1 and 2 as to invert positions 50 and 51.

Listwise: optimise the full list simultaneously. LambdaMART directly optimises NDCG. SoftRank and ListNet use differentiable approximations. These are the highest-performing approaches.

LambdaRank: the practical solution

LambdaRank (Burges et al., 2006) trains a neural network ranking model by defining "lambda gradients" — gradient magnitudes that are heuristically motivated to correlate with NDCG improvement. For a pair (i, j) with i more relevant: λ_ij = |ΔNDCG_ij| * σ(-s_ij). The |ΔNDCG_ij| term is the change in NDCG if you swap items i and j's positions — it weights pairs by how much swapping them would hurt the ranking. The σ(-s_ij) term is the standard pairwise gradient. These λ gradients are not derived from any loss function, but training with them directly improves NDCG. LambdaMART applies these gradients within a gradient boosted tree framework and is still competitive with neural approaches on many benchmarks.

Feature engineering for ranking

A ranking model typically has three types of features: query features (query text embedding, query frequency, query category), document features (document text embedding, historical CTR, recency, quality score), and interaction features (query-document similarity, user-document co-engagement, dwell time on previous encounters). The interaction features are the most powerful — they encode how specifically this user and this document have interacted before.

Calibration and business rules

A ranking model's raw scores are not probabilities and do not have a natural scale. In production, re-ranking layers apply business rules on top of the model score: boost content from premium partners, penalise repetitive content, apply diversity constraints (no more than two items from the same creator in the top 10). The model provides relevance signal; business logic shapes the final list.

Try on Colab: use the LETOR dataset (Microsoft Learning to Rank benchmark). Train three models: a pointwise regression, a pairwise RankNet, and a LambdaMART (use LightGBM's rank objective). Evaluate all three on NDCG@5 and NDCG@10. Plot the performance difference. The listwise model should outperform pointwise by 2-5 NDCG points.