Ranking Metrics

NDCG, MAP, MRR — search and recommendation quality

You type "Python tutorial for beginners" into a search box and get back ten results. Suppose the truly useful ones sit at positions 1, 3, and 7. A metric that just counts "3 of 10 are relevant" scores this 0.30 — and would give the *same* 0.30 if those three results were buried at positions 8, 9, and 10. But you know that is wrong: nobody reads past the first couple of results once they have their answer. Any metric that ignores *position* is nearly useless for search and recommendation.

So the whole family of ranking metrics is built on one idea: a relevant result near the top is worth far more than the same result near the bottom. They differ in how much detail they capture.

MRR — where is the first good one?

The simplest is Mean Reciprocal Rank. It cares about one thing: the position of the *first* relevant result. If it is at position 1 you score 1/1 = 1.0; at position 3, 1/3 = 0.33. Average that across all your queries and you have MRR. Flip it around and 1/MRR reads as the *effective rank* of the first relevant result: MRR = 0.60 means the first hit lands, on average, around rank 1.67; push MRR to 0.75 and that effective rank drops to about 1.33. It fits perfectly when the user has a single need and stops at the first good answer — a fact lookup, an FAQ, a "just take me to X" query. It is the wrong choice the moment users want *several* relevant results, because it ignores everything after the first hit.

MAP — did you find them all, and early?

Mean Average Precision rewards surfacing relevant results both early *and* completely. For one query, walk down the list, and each time you hit a relevant result note the precision so far; sum those, then divide by the total number of relevant results. For our example (relevant at 1, 3, 7, three relevant total): (1/3)(1/1 + 2/3 + 3/7) ≈ 0.70. Average across queries to get MAP. It only needs yes/no relevance labels, which makes it cheap to collect.

NDCG — the industry standard

Normalized Discounted Cumulative Gain adds the one thing MAP lacks: *grades* of relevance. A result can be perfect (grade 3), okay (grade 1), or useless (grade 0), not just relevant or not. NDCG folds in two moves: each result contributes (2^grade − 1), so a perfect hit counts far more than a so-so one; and each position is discounted by log₂(position + 1), so position 1 counts fully, position 3 about half, position 10 under a third. Divide by the score of the *ideal* ordering and you get a number from 0 to 1 that is comparable across queries. This is what commercial search and recommendation teams actually use, when they have graded human judgments and position matters.

Which one? Match the metric to what users do

There is no single best ranking metric — the right one mirrors real behavior. If users scan a vertical list and stop at the first hit, use MRR. If they expect to find all the relevant results and would be annoyed to see any buried, use MAP. If you have graded labels and care about exact ordering, use NDCG. And if every shown slot is equally prominent regardless of order — a grid of products, a row of ad slots — plain Precision@K is fine. Pick by what the interface makes people do, not by which formula looks most impressive.

Retrieval's own metrics: recall@K and hit-rate@K

Big systems rank in two stages — a cheap candidate generator pulls a few hundred items from millions, then an expensive ranker orders them. These stages need different metrics. For retrieval the question isn't "is the order perfect?" but "did we even *retrieve* the relevant items into the candidate set?" That's recall@K (of all relevant items, how many made it into the top K) and hit-rate@K (did at least one relevant item land in the top K). A ranker can't fix what retrieval never surfaced, so you measure recall@K on the generator and NDCG/MAP on the ranker — diagnosing the wrong stage is a classic mistake.

Relevance isn't the only goal

Pure relevance metrics miss things a real recommender must balance. Coverage — what fraction of the catalog ever gets shown (a system that only recommends the top 100 items starves the long tail). Diversity — are the top results varied, or ten near-duplicates? Novelty / serendipity — does it surface things the user wouldn't have found alone, not just the obvious? Freshness — new content shown before it goes stale. Creator fairness — is exposure spread across creators or concentrated? A model that maxes NDCG by always showing the same popular items can quietly wreck coverage and diversity, and the business notices even though the relevance metric looks great.

The elephant: position bias in click labels

If your relevance labels come from *clicks*, they're contaminated. The item at position 1 gets more clicks because it was shown first, not because it's more relevant — so training or evaluating on raw clicks teaches the model to reproduce the old ranking's position effects rather than true relevance, a self-reinforcing loop. The fixes are their own field: inverse-propensity weighting to down-weight clicks that only happened due to position, interleaving (blend two rankers' results and see which gets clicked, which cancels position bias), and counterfactual evaluation — estimating how a *different* ranking policy would have performed, using data logged under the old policy, without actually deploying the new one to find out. Never treat click data as clean relevance.

Offline-online gaps: it isn't always position bias

Position bias in click labels is not the only reason a healthy offline score can miss what users feel. The offline test set itself can be stale — judgments and even the candidate catalog were captured at some past date, so a model can keep matching outdated ground truth while the real catalog and real user intent have moved on. And offline metrics are almost always computed per item — one score per query, per result list — while the actual user experience unfolds per session, across several searches or scroll loads in a row; a ranking that scores fine query-by-query can still feel repetitive or exhausting once you look at the whole session. When an offline number and online sentiment disagree, check the freshness of your judgments and whether you're measuring the right unit (item vs. session) before assuming the metric itself is wrong.

Metric@K: K is a product decision

NDCG@1, @3, @10, @50 answer *different* questions, and K should match how the interface is actually used. NDCG@1 is for "I feel lucky" single-answer surfaces; NDCG@10 for a first page of ten; NDCG@50 for a scroll-heavy feed or a candidate pool feeding a downstream reranker. Set K to the viewport and session behaviour, not to a default — reporting NDCG@10 for a mobile screen that shows three results is measuring the wrong thing.

Ties and unjudged documents

Two practical hazards. Ties (items with equal score) make the metric depend on arbitrary tiebreak order — resolve them deterministically or you'll see phantom metric swings. And unjudged ≠ irrelevant: with millions of documents, only a pooled subset gets human labels, so a genuinely relevant document nobody judged is scored 0 by default, penalising a new system that surfaces it. This pooling bias is why a better retriever can look worse offline; judge the novel results before trusting the comparison.

You can't optimise these metrics directly

A final subtlety interviewers probe: NDCG, MAP, and MRR are based on *sorting*, which is not differentiable, so you can't gradient-descend on them directly. Ranking models therefore optimise a surrogate loss — pairwise (learn "A should rank above B," as in RankNet/LambdaRank) or listwise (score the whole list, as in ListNet) — that correlates with the target metric while being differentiable. LambdaMART's trick is to weight those pairwise gradients by their NDCG impact, getting a ranking-aware signal without needing NDCG to be differentiable.

Key points

• Match the metric to the user's behavior, not to mathematical elegance. Graded labels and position matters (commercial search)? Use NDCG. User wants one answer and stops at the first hit (question-answering, lookups)? Use MRR. Every shown slot equally prominent regardless of order (a product grid, an ad row)? Use Precision@K. Choose the wrong one and you optimise a proxy that does not track the real experience — NDCG@10 can even climb while MRR falls, if the model improves positions 4–10 while making position 1 worse.
• The trap: judging against an incomplete set of relevance labels. Raters can only score a small pool of documents out of millions. If that pool was built from the old system's top results, a new system that surfaces genuinely relevant documents nobody ever judged will see them scored as "not relevant" by default — so it looks worse on NDCG even though it found better results. When a change is big (a new retrieval or ranking model), judge the new system's novel results before comparing scores.
• The diagnostic: look at NDCG@1 versus NDCG@10 separately. If NDCG@10 is healthy but NDCG@1 is much lower — say NDCG@10 = 0.90 but NDCG@1 = 0.45 — the relevant results are in the top ten but the best one is not landing first — your re-ranking is the bottleneck, not retrieval. If NDCG@1 and NDCG@10 are both low and roughly equal — say NDCG@1 = 0.20 and NDCG@10 = 0.25 — the relevant documents are not even in the candidate set — retrieval is the bottleneck. The split tells you which stage to go fix.
• Measure the right stage with the right metric, and don't trust raw click labels. Candidate generation is judged by recall@K / hit-rate@K (did the relevant items make it into the pool?), the ranker by NDCG/MAP — diagnosing the wrong stage wastes weeks. Set K to the actual viewport (NDCG@1/@3/@10/@50 answer different product questions). And clicks are position-biased: the top slot gets clicks because it was shown first, so debias with inverse-propensity weighting, interleaving, or counterfactual evaluation before treating clicks as relevance.
• Relevance isn't the whole story, and you can't optimise these metrics directly. Balance NDCG against coverage, diversity, novelty/serendipity, freshness, and creator fairness — a model that maxes relevance by always showing the same popular items wrecks the catalog and long tail. Handle ties deterministically and remember unjudged ≠ irrelevant (pooling bias makes a better retriever look worse offline). Since NDCG/MAP/MRR rely on non-differentiable sorting, ranking models train on pairwise (RankNet/LambdaRank) or listwise (ListNet) surrogate losses — LambdaMART weights pairwise gradients by NDCG impact to stay ranking-aware.

Recap

• Any metric that ignores position is useless for search/recsys: relevant results at positions 1, 3, 7 score the same "3 of 10" as results buried at 8, 9, 10 — but nobody reads past the first couple. The whole family is built on one idea: a relevant result near the top is worth far more than the same result near the bottom.
• MRR = 1/rank of the *first* relevant result: position 1 → 1.0, position 3 → 0.33, averaged across queries. It fits when the user has one need and stops at the first good answer (fact lookup, FAQ, "just take me to X") — wrong the moment users want several relevant results.
• MAP rewards finding them all, and early: walk down the list, note precision-so-far at each relevant hit, average those, divide by total relevant. For relevant at 1, 3, 7 (three total): (1/3)(1/1 + 2/3 + 3/7) ≈ 0.70. Needs only binary yes/no labels, so it's cheap to collect.
• NDCG is the industry standard because it adds *graded* relevance: each result contributes $(2^{grade}-1)$ gain (a perfect hit counts far more than a so-so one) discounted by $\log_2(pos+1)$, divided by the ideal ordering's score for a 0–1 number comparable across queries. Use it when you have graded human judgments and exact ordering matters.
• Match the metric to user behaviour, not to which formula looks impressive: scan-and-stop → MRR; expect to find all relevant items → MAP; graded labels + care about order → NDCG; every shown slot equally prominent (a product grid, an ad row) → plain Precision@K.
• Two stages need two different metrics: a cheap candidate generator is judged by recall@K / hit-rate@K (did the relevant items even reach the candidate set?), the expensive ranker by NDCG/MAP — a ranker can't fix what retrieval never surfaced, so diagnosing the wrong stage wastes weeks.
• Watch three traps: click labels are position-biased (top slot gets clicks because it was shown first — debias with IPW, interleaving, or counterfactual eval); pure relevance ignores coverage/diversity/novelty/freshness/creator-fairness (maxing NDCG by always showing popular items wrecks the catalog); and NDCG/MAP/MRR rely on non-differentiable sorting, so rankers train on pairwise (RankNet/LambdaRank) or listwise (ListNet) surrogates — LambdaMART weights pairwise gradients by NDCG impact.

Check your understanding