Significance Testing in IR Evaluation

A new ranker scores MAP 0.312 against the baseline's 0.297 on 50 TREC topics. The per-topic differences have a standard deviation of 0.08. The paired t statistic is

which with 49 degrees of freedom gives a two-sided \(p\) of about 0.19. The improvement is not distinguishable from the luck of which 50 topics were written. Had the per-topic differences been twice as consistent (\(s_d = 0.04\)), the same mean gap would give \(t \approx 2.65\) and \(p \approx 0.01\). A mean gap means little until you know its topic-to-topic spread.

Topics are the sample

The unit of analysis in batch IR evaluation is the topic. For systems \(A\) and \(B\) and topics \(i = 1, \dots, n\), the data are differences \(d_i = s_A(i) - s_B(i)\) in some per-topic metric, and the null hypothesis is that the mean difference over the population of topics like these is zero. Pairing matters enormously: variance between topics (some queries are easy for every system) dwarfs variance between systems, and an unpaired test spends its power measuring topic difficulty.

Three families of test are in common use.

The paired t-test uses the statistic above and assumes the mean difference is approximately normal, which the central limit theorem supplies surprisingly well at \(n = 50\) even when per-topic scores are bounded and skewed.

The randomisation (permutation) test makes no distributional assumption. Under the null, the labels \(A\) and \(B\) are exchangeable within each topic, so each \(d_i\) is equally likely to have either sign. Enumerate or sample sign flips \(\sigma_i \in \{-1, +1\}\) and compute

With 50 topics there are \(2^{50} \approx 1.1 \times 10^{15}\) assignments, so in practice one samples \(10^4\) to \(10^5\) of them.

The bootstrap resamples topics with replacement from the shifted differences \(d_i - \bar d\) and asks how often a resampled mean is as extreme as the observed one.

What the comparisons found

Smucker, Allan and Carterette ran all pairs of ad hoc runs from TRECs 3 and 5 to 8 through five tests. The randomisation test, the bootstrap and the t-test produced comparable p-values; the Wilcoxon signed-rank and sign tests produced very different ones, and the authors recommended discontinuing those two and using the randomisation test (Smucker, Allan and Carterette, 2007, A Comparison of Statistical Significance Tests for Information Retrieval Evaluation, CIKM).

Their study could compare tests with each other but could not know the true answer. Urbano, Lima and Hanjalic got around that by simulating new systems from TREC data with known null hypotheses, computing over 500 million p-values. The t-test and permutation test held the Type I error rate at the nominal level across metrics and topic set sizes; the bootstrap-shift test was biased toward small p-values; Wilcoxon and sign tests were unreliable. Their top recommendation was the t-test, for its simplicity and robustness to sample size (Urbano, Lima and Hanjalic, 2019, Statistical Significance Testing in Information Retrieval: An Empirical Analysis of Type I, Type II and Type III Errors, SIGIR, arXiv:1905.11096).

The disagreement is small but real. The 2007 paper preferred the randomisation test on principle; the 2019 study found it and the t-test nearly indistinguishable and chose the simpler one. Both agree against the textbook advice that Wilcoxon is the safe non-parametric choice for non-normal scores, a point Urbano pressed further at SIGIR 2026 with a paper arguing that Wilcoxon routinely loses control of its Type I error on IR data (Urbano, 2026, Stop Using the Wilcoxon Test: Myth, Misconception and Misuse in IR Research, arXiv:2604.25349).

Many comparisons at once

A paper that compares 20 variants against a baseline, each at \(\alpha = 0.05\), expects about one false positive even if nothing works: the familywise error rate is \(1 - 0.95^{20} \approx 0.64\). Bonferroni correction tests each at \(\alpha/m = 0.0025\), which controls that rate and costs power. Carterette modelled IR experiments as the simultaneous tests they are, sharing topics and a reused collection, and found that accounting for these sources of randomness can raise p-values by orders of magnitude, to the point where it becomes "very difficult to conclude that anything is significant" (Carterette, 2012, Multiple Testing in Statistical Analysis of Systems-Based Information Retrieval Experiments, ACM TOIS 30(1)).

The reporting picture is not reassuring. Sakai reviewed 840 SIGIR full papers and 215 TOIS papers from 2006 to 2015 and found that many effectiveness papers either did no significance testing or did not report p-values or test statistics, which made it impossible to check whether experiments were underpowered (Sakai, 2016, Statistical Significance, Power, and Sample Sizes: A Systematic Review of SIGIR and TOIS, 2006-2015, SIGIR).

When it breaks

Significance is not size. On a collection with thousands of queries, a difference too small for any user to notice will be significant. Report a confidence interval for \(\bar d\) and the count of topics improved and hurt.

The test generalises over topics only. The population being sampled is topics like these, judged by these assessors, over this corpus. A significant result says nothing about a different document collection or assessor pool, and reusing the same collection for years turns the whole field into one long multiple-comparisons problem.

Choosing the test after seeing the results is p-hacking. Running three tests and reporting whichever crosses 0.05 inflates the error rate. Fix the test, metric and cutoff before looking.

Non-significant does not mean equivalent. With 50 topics and a noisy metric, power to detect a real 2% gain can be low. Showing that two systems are equivalent needs an equivalence test with a stated margin, not a failed difference test.