Search Evaluation
Pooling and judgments, nDCG and MRR, interleaving, online metrics, and why offline gains vanish online.
7concepts
129flashcards
51minutes of reading
- 01 Precision, Recall and Average Precision The binary-relevance metric family that underpins ranked retrieval evaluation, from set precision and recall through P@k, R-precision and interpolated curves to average precision, and the user model each one quietly assumes.
- 02 Test Collections, Pooling and Judgment Bias The Cranfield paradigm made retrieval a measurable science, and the pooling shortcut that makes it affordable quietly penalises any system unlike the ones that built the pool.
- 03 nDCG, MRR and Graded Relevance The main ranking metrics differ in what they assume about the user, and choosing one is choosing a model of how far someone reads and what they are looking for.
- 04 Interleaving and Online Evaluation Mixing two rankers' results into a single list and attributing clicks gives a within-user paired comparison that detects differences far faster than an A/B test on the same traffic.
- 05 LLM Relevance Judgments Using large language models to label query-document relevance for offline evaluation, what the agreement numbers actually show, and the circularity that appears once the systems being judged use the same kind of model as the judge.
- 06 Significance Testing in IR Evaluation How to decide whether a MAP or nDCG difference over a few dozen topics is signal, which paired tests actually hold their error rates on IR data, and why testing many variants at once quietly inflates what counts as significant.
- 07 Why Offline Gains Vanish Online The recurring experience that an offline nDCG improvement produces no measurable online effect, and the four distinct mechanisms that cause it.