nDCG, MRR and Graded Relevance

Every ranking metric encodes a user model. Reciprocal rank assumes the user wants one answer and stops at the first relevant result. Precision@10 assumes they scan ten results and weight them equally. nDCG assumes graded usefulness and a smoothly decaying chance of looking further. None is more correct in general; the question is which matches the task.

The metrics and their assumptions

Precision@k is the fraction of the top \(k\) that is relevant. It is flat: a relevant document at rank 1 and at rank 10 count identically, which contradicts everything known about how people read result lists. Its virtue is that it is impossible to misunderstand.

MRR is \(1/\mathrm{rank}\) of the first relevant result, averaged over queries. It models a user seeking one answer, so it fits navigational and question-answering tasks and ignores everything after the first hit. On a query with twenty relevant documents, MRR cannot tell a system that found all twenty from one that found one.

MAP averages precision at each relevant document's rank, rewarding both finding relevant documents and placing them early. It assumes binary relevance.

nDCG is the one that handles graded labels (Järvelin and Kekäläinen, 2002, Cumulated Gain-based Evaluation of IR Techniques, ACM TOIS 20(4), 422-446):

normalised by the DCG of the ideal ordering. Two design choices carry the meaning. The exponential gain \(2^{\mathrm{rel}} - 1\) makes a "perfect" (grade 3) document worth 7 while a "fair" (grade 1) is worth 1, so highly relevant documents dominate; a linear gain would make three fair documents equal to one perfect one. The logarithmic discount decays slowly, encoding a user who keeps reading with gradually declining probability.

Why normalisation matters and what it hides

Dividing by the ideal DCG puts every query on \([0,1]\), which is what makes averaging across queries meaningful: without it, a query with forty relevant documents would dominate the mean over one with two.

It also hides difficulty. A query where the system achieves nDCG 0.9 against an ideal containing one relevant document, and one where it achieves 0.9 against an ideal containing forty, are reported identically. And a query with no relevant documents at all has an undefined normaliser, which implementations handle differently, some assigning 0 and some excluding the query, which is a real source of disagreement between two tools computing "the same" metric.

When it breaks

Cutoff choice is a product decision disguised as a parameter. nDCG@10 measures what fits above the fold; nDCG@100 measures candidate-set quality for a reranker. A change can improve one and hurt the other, and reporting only the flattering cutoff is common enough to be worth checking for.

Grade-to-gain mapping is arbitrary. The exponential form is convention, not derivation. With a four-point scale, whether "perfect" is worth 7 times "fair" or 3 times changes which system wins in close comparisons, and papers rarely state the mapping used.

No metric here models diversity or redundancy. Ten identical relevant documents score the same as ten different ones covering distinct aspects. For ambiguous or exploratory queries this is badly wrong, and \(\alpha\)-nDCG and intent-aware variants exist precisely because the standard metric is blind to it.

Position discounts are not examination probabilities. The \(1/\log_2(i+1)\) discount is a modelling convention chosen for its properties, and it does not match measured examination curves, which are steeper at the top. Metrics derived from measured user behaviour, such as expected reciprocal rank or time-based measures, fit observed behaviour better and are less standard.

Averaging over queries hides distribution. A mean nDCG improvement of 0.02 can be a uniform small gain or a large gain on 10% of queries and a loss on 30%. The second is a much riskier launch and looks identical in the summary.