Search Evaluation
Pooling and judgments, nDCG and MRR, interleaving, online metrics, and why offline gains vanish online.
7concepts
129flashcards
51minutes of reading
- 01 Interleaving and Online Evaluation Mixing two rankers' results into a single list and attributing clicks gives a within-user paired comparison that detects differences far faster than an A/B test on the same traffic.
- 02 LLM Relevance Judgments Using large language models to label query-document relevance for offline evaluation, what the agreement numbers actually show, and the circularity that appears once the systems being judged use the same kind of model as the judge.
- 03 Significance Testing in IR Evaluation How to decide whether a MAP or nDCG difference over a few dozen topics is signal, which paired tests actually hold their error rates on IR data, and why testing many variants at once quietly inflates what counts as significant.
- 04 Why Offline Gains Vanish Online The recurring experience that an offline nDCG improvement produces no measurable online effect, and the four distinct mechanisms that cause it.