Error Bars for Evals: Sampling, Clustering and Paired Tests

Two models score 71.2 and 69.8 on a 250-question benchmark. A launch post calls the first "state of the art". The standard error on each score is about 2.9 points, the standard error on the difference is larger still, and the honest reading is that the benchmark cannot tell these models apart.

Evaluations are experiments. The correct frame, argued in the paper that finally made this mainstream in the LLM world, is that the questions in your eval are a sample from a super-population of questions you could have asked (Miller, 2024, Adding Error Bars to Evals, arXiv:2411.00640). You do not care about the 250 questions. You care about the task they represent, and that inference has uncertainty.

The basic interval

For a binary-scored eval with \(n\) independent questions and observed mean accuracy \(\bar{x}\), the standard error of the mean is

At \(\bar{x} = 0.70\) this gives 2.9 points for \(n = 250\), 1.4 for \(n = 1{,}000\), and 0.46 for \(n = 10{,}000\). The \(\sqrt{n}\) is unforgiving: to halve your interval you need four times the questions. Miller's practical guidance falls straight out of this, that evals need on the order of 1,000 questions or more to have useful power against the effect sizes people actually report.

Two common benchmarks fail this test outright. GPQA Diamond has 198 questions (Rein et al., 2023, arXiv:2311.12022); AIME has 15 per year. A one-question swing on AIME moves the headline number by 6.7 points.

Clustering, the mistake almost everyone makes

The formula above assumes independent questions. Benchmarks are rarely built that way. MMLU groups questions by subject. SWE-bench draws multiple issues from the same repository. A reading-comprehension set asks five questions about one passage.

Within a cluster, outcomes are correlated: if the model does not know virology, it misses the whole virology block together. Treating \(n\) as the number of questions rather than the effective number of independent units understates variance. The clustered standard error is computed over cluster-level means, and the inflation is not marginal; Miller reports that accounting for clustering can inflate standard errors by up to a factor of three relative to the naive calculation. A three-times-too-narrow confidence interval is how a null result gets published as a win.

Pair everything

The highest-return change costs nothing. When comparing two models, do not compute two independent intervals and check whether they overlap. Compute the per-question difference and put the interval on that.

Both models face the same questions, so question difficulty, the dominant variance component, cancels. Let \(d_i = x_i^{A} - x_i^{B}\); then

Miller reports that exploiting the question-level correlation this way reduces variance by roughly a third in relative terms. Note also that non-overlapping intervals imply significance but overlapping ones do not imply the reverse, so the eyeball test on two error bars is biased toward "no difference".

Reducing variance you control

Two variance sources are in play: sampling questions, and sampling the model's own output. The second is yours to manage.

Generating \(K\) answers per question and averaging shrinks the within-question component by \(1/K\) but leaves the between-question component untouched, so returns saturate quickly; past \(K \approx 5\) you are usually better off buying more questions. Where a benchmark is multiple choice, using next-token probabilities instead of sampled answers removes the decoding variance entirely.

What does not work is lowering temperature to make numbers look stable. It reduces variance by changing the estimand, so you are now reporting greedy-decode performance, which is a different quantity from what users experience at temperature 1.

When it breaks

Statistical significance is not practical significance. With 20,000 questions a 0.3-point gap becomes significant and remains irrelevant. Decide the minimum effect size you care about before running, and size the eval to detect it.

Multiple comparisons are everywhere. Reporting 15 benchmarks and highlighting the three wins is a garden of forking paths. At \(\alpha = 0.05\) and 15 independent tests, the chance of at least one false positive is 54 percent.

The super-population may not exist. The framework assumes your questions are exchangeable draws from a population you care about. For a hand-curated adversarial set, they are not, and a confidence interval over that sample answers a question nobody asked.