Test Collections, Pooling and Judgment Bias

A test collection is three things: a corpus, a set of queries, and relevance judgments saying which documents are relevant to which query. Given all three, comparing two ranking systems is arithmetic. This is the Cranfield paradigm, and it is why information retrieval has had reproducible offline evaluation since the 1960s while many adjacent fields did not.

The problem is the third component. Judging one query against a million documents is impossible, and judging 50 queries against it is impossible fifty times over.

Pooling, and the bias it introduces

The TREC solution is pooling: run many systems on each query, take the top \(k\) results from each, form the union, and judge only that pool. Every document nobody's system returned is assumed irrelevant. With enough diverse participating systems, the pool captures most of the genuinely relevant documents, and the assumption is approximately harmless.

Approximately. The bias is specific and directional: a new system that finds relevant documents no pooled system found is penalised, because those documents sit in the unjudged set and are scored as irrelevant. The evaluation systematically favours systems resembling those that built the pool.

This matters most for exactly the systems worth evaluating. When dense retrieval arrived, it retrieved semantically relevant documents sharing no vocabulary with the query, precisely the documents a pool built from lexical systems would not contain. Some early dense-versus-BM25 comparisons on older collections were measuring pool composition as much as retrieval quality, and re-judging efforts on MS MARCO and TREC Deep Learning found substantial numbers of unjudged relevant documents returned by newer systems.

Pool depth is the tunable defence: deeper pools cost linearly more judging and cover more. bpref and infAP are metrics designed to be robust to incomplete judgments by ignoring or estimating over unjudged documents, and they are the right choice when reusing an old collection with a new class of system.

Sparse labels and their consequences

MS MARCO, the collection most retrieval models are now trained and evaluated on, has roughly one judged relevant passage per query, derived from what a human found useful rather than from exhaustive judging. That makes it enormous and cheap, and it makes MRR@10 on it a measurement of whether the system found that particular passage, not whether it found a good one.

Two systems can differ meaningfully on MS MARCO while being indistinguishable to a user, and a system can be penalised for returning a better passage than the labelled one. This is understood in the field and routinely forgotten when a leaderboard number is quoted.

When it breaks

Queries are not a random sample of anything. Fifty TREC topics were written by assessors; MS MARCO queries came from a specific search engine's traffic in a specific period. Neither represents your users' queries, and the ordering of two systems can genuinely reverse on a different query distribution.

Relevance is treated as a property of a document. It is a property of a document, a user, a task and a moment. Assessor agreement on binary relevance is far from perfect, which puts a ceiling on how finely any collection can discriminate. The classic reassurance is that relative system orderings are fairly stable across assessors even when absolute scores are not, and that reassurance is about ordering, not about the size of a difference.

Contamination is now a first-order concern. Retrieval models trained on web-scale corpora may have seen the test collection's documents and its queries, so an improvement can reflect memorisation. This is the same problem LLM benchmarks face, arriving in a field whose collections have been public for decades.

Statistical significance is under-reported. With 50 queries, a 2% nDCG difference is frequently within noise, and per-query variance in retrieval is large. Paired tests across queries and reporting the number of queries improved versus hurt are more informative than the mean difference alone.