Error Bars for Evals: Why Most Benchmark Differences Are Noise

GPQA Diamond has 198 questions (Rein et al., 2023, arXiv:2311.12022). AIME has fifteen per year. A single question on AIME is worth 6.7 percentage points, which means a model that gets lucky on one problem beats a model that did not by more than the gap between most competing frontier releases. These are two of the benchmarks that anchor frontier model launches, and neither can resolve the differences it is routinely used to report.

The problem is not that the benchmarks are small. It is that scores from them are published as if they were measurements of a fixed quantity, when they are estimates from a sample, with all the uncertainty that implies. Evan Miller made the argument precisely in late 2024: an evaluation is an experiment, and the questions in it are a sample from a super-population of questions you could have asked (Miller, 2024, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations, arXiv:2411.00640). Nobody cares about those 198 questions. They care about graduate-level science reasoning, and that inference has a standard error.

Why this matters: Every model selection decision, every fine-tuning iteration, and every "state of the art" claim rests on comparing two numbers. If the difference between those numbers is smaller than the noise in either of them, the comparison is a coin flip dressed as evidence, and the resources spent acting on it are wasted.

TL;DR

• The standard error of a binary-scored eval at 70 percent accuracy is 2.9 points for \(n = 250\), 1.4 for \(n = 1{,}000\), and 0.46 for \(n = 10{,}000\). Halving the interval costs four times the questions.
• Miller's practical guidance is that evals need on the order of 1,000 questions or more for useful power against realistic effect sizes. GPQA Diamond has 198; SWE-bench Verified has 500.
• Clustering is the most under-corrected error in the field. Questions grouped by subject or repository are correlated, and ignoring that can inflate the true standard error by up to a factor of three.
• Paired analysis costs nothing and buys roughly a third of the variance back, because both models face the same questions and question difficulty cancels.
• Comparing two independent error bars by eye is biased toward "no difference": non-overlapping intervals imply significance, but overlapping ones do not imply its absence.
• Benchmarks also carry systematic error that no sample size fixes. An estimated 6.49 percent of MMLU questions contain ground-truth errors, rising to 57 percent of the analysed Virology subset (Gema et al., NAACL 2025, arXiv:2406.04127).
• Reporting fifteen benchmarks and highlighting three wins carries a 54 percent chance of at least one false positive at \(\alpha = 0.05\), before anyone has cheated.
• The fix is boring and cheap: pair, cluster-correct, power-analyse before running, and pre-register which benchmarks are the headline.

At a Glance

Two models, one benchmark, four sources of uncertainty that stack. Only the last two get discussed.

flowchart LR
  Q["Question sample<br/>n questions"] --> S["Sampling error<br/>SE ~ 1/sqrt(n)"]
  C["Cluster structure<br/>subjects, repos"] --> S
  D["Decoding randomness<br/>temperature, K samples"] --> S
  L["Label errors<br/>systematic, not shrinking"] --> B["Reported score"]
  S --> B
  B --> V["Is the gap real?"]
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
  class Q,C,D blue
  class S amber
  class L rose
  class B,V teal

[IMAGE: Two overlapping normal distributions centred at 69.6 percent and 71.2 percent, each with standard deviation 2.9 points, with the overlap region shaded. Caption: "The two distributions a 250-question benchmark actually reports. The published headline is the distance between the two peaks."]

How Evaluation Got Here

Benchmark culture in NLP grew out of a setting where the statistics did not matter much. GLUE and SuperGLUE were leaderboards over task suites with tens of thousands of test items, and the differences between submissions were large (Wang et al., 2018, arXiv:1804.07461). With \(n\) in the tens of thousands and gaps in the tens of points, nobody needed a confidence interval to see who won.

Then the field changed shape three times in five years. Benchmarks got harder and therefore smaller, because writing a graduate-level physics question that experts disagree about is expensive and writing a sentiment label is not. Models got closer together, because everyone trains on similar data with similar recipes. And the reporting became commercial: benchmark tables moved from the results section of a paper to the launch post of a product.

Small \(n\), small effects, high stakes. That is the exact configuration in which statistical discipline stops being optional, and it arrived without the discipline.

timeline
    title From leaderboards to launch posts
    2018 : GLUE ships nine tasks with large test sets and large gaps
    2020 : MMLU introduces 15,908 questions over 57 subjects and becomes the default headline number
    2022 : HELM evaluates 42 scenarios over seven metric categories, arguing one number is not enough
         : BIG-bench collects 200-plus tasks from 400-plus authors
    2023 : GPQA ships 198 Diamond questions; SWE-bench ships 2,294 execution-graded issues
         : Chatbot Arena opens pairwise human voting as an alternative to static sets
    2024 : SWE-bench Verified cuts to 500 human-validated instances after annotators find broken tasks
         : MMLU-Redux finds 6.49 percent ground-truth error; Miller publishes Adding Error Bars to Evals
    2025 : The Leaderboard Illusion documents private pre-testing and selective disclosure on Chatbot Arena

Each of those 2024 and 2025 entries is the same discovery arriving from a different direction: the number on the leaderboard has more uncertainty in it than the leaderboard admits.

[IMAGE: Dual-axis chart over 2018 to 2025. Left axis, plotted as bars: median test-set size of the benchmark headlining frontier releases each year, falling from tens of thousands to hundreds. Right axis, plotted as a line: median reported gap between the top two models on that benchmark, also falling. Caption: "Both curves went down together, which is why the statistics started to matter."]

The Statistics an Eval Report Needs

The interval

Score a binary-graded eval with \(n\) independent questions, observed mean \(\bar{x}\). The estimator is a sample mean of Bernoulli outcomes, so

and the 95 percent interval is \(\bar{x} \pm 1.96 \times \mathrm{SE}\). At \(\bar{x} = 0.70\):

The \(\sqrt{n}\) is the whole story. Going from 250 to 1,000 questions halves the interval; getting to a tenth of a point needs six figures of questions. This is why Miller's recommendation lands where it does, and why the small hard benchmarks that dominate model launches are structurally unable to support the claims made on them.

Clustering, the mistake nearly everyone makes

The formula above assumes independence. Almost no benchmark is built that way.

MMLU groups questions by subject: 57 blocks, and a model that has not learned virology misses that block together. SWE-bench draws multiple issues from each of a small number of repositories, and a model that cannot navigate Django's ORM fails several instances for the same reason. Reading comprehension sets ask several questions about one passage.

Within a cluster, outcomes are positively correlated, so the effective number of independent observations is closer to the number of clusters than to the number of questions. The standard correction computes variance across cluster means rather than across items. The magnitude is not a rounding detail: Miller reports that accounting for clustering can inflate standard errors by up to a factor of three relative to the naive calculation.

A useful way to think about it is the design effect. For \(m\) items per cluster with intra-cluster correlation \(\rho\),

and the corrected standard error is \(\sqrt{\mathrm{DEFF}}\) times the naive one. Ten items per cluster at \(\rho = 0.15\) gives \(\mathrm{DEFF} = 2.35\), so the true interval is 1.53 times wider than reported. That factor alone converts a large fraction of published "significant" gaps into ties.

Pair everything

Here is the change that costs nothing and returns the most.

When comparing two models, the wrong procedure is to compute two independent intervals and check whether they overlap. The right one is to compute the per-question difference and put the interval on that. Let \(d_i = x_i^{A} - x_i^{B} \in \{-1, 0, +1\}\) for a binary-graded item. Then

Both models faced the same questions, so question difficulty, which is the dominant variance component in almost every eval, cancels exactly. The \(d_i\) are zero on every question where the models agree, and agreement between two frontier models on the same benchmark is typically high. Miller reports variance reductions of roughly a third in relative terms from exploiting this correlation.

There is a second reason to pair, which is that the eyeball test on two independent intervals is biased. Non-overlapping intervals do imply a significant difference. Overlapping intervals do not imply the absence of one, because the standard error of a difference of independent means is \(\sqrt{\mathrm{SE}_A^2 + \mathrm{SE}_B^2}\), which is smaller than \(\mathrm{SE}_A + \mathrm{SE}_B\). Reading two error bars by eye systematically under-detects real differences while the naive point comparison over-detects them, which is an unusually unhelpful combination.

The variance you control

Two distinct sources of randomness are in play, and only one is a property of the benchmark.

Between-question variance comes from which questions you happened to sample. The only fix is more questions.

Within-question variance comes from the model's own sampling. Generating \(K\) answers per question and averaging the score shrinks this component by \(1/K\) while leaving the between-question component untouched. Because the between-question term usually dominates, returns saturate fast: past roughly \(K = 5\), the marginal dollar is better spent on more questions than on more samples of the same question. Where the task is multiple choice, reading next-token probabilities removes decoding variance entirely at no cost.

What does not count is turning the temperature down to make the numbers stable. It reduces variance by changing the estimand: you are now reporting greedy-decode performance, which is a different quantity from the temperature-1 performance users experience. Miller lists this explicitly as something not to do, and it remains common.

Seeing It in Motion

The decision procedure, drawn as it should be run:

flowchart TB
  A["Define minimum effect<br/>size worth detecting"] --> B["Power analysis:<br/>how many questions?"]
  B --> C{"Do you have<br/>that many?"}
  C -->|No| D["Report a tie, or<br/>build more questions"]
  C -->|Yes| E["Run both models<br/>on identical items"]
  E --> F["Compute per-question<br/>differences d_i"]
  F --> G{"Clustered<br/>items?"}
  G -->|Yes| H["Cluster-robust SE<br/>over cluster means"]
  G -->|No| I["Standard SE"]
  H --> J["Report difference,<br/>CI and p-value"]
  I --> J
  classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
  classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
  classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
  classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
  class A,B blue
  class C,G purple
  class D rose
  class E,F,H,I purple
  class J emerald

And the same thing as a lifecycle, because the failure is usually one of ordering: teams size the eval after seeing the result they want.

stateDiagram-v2
    [*] --> Specified: name the effect size
    Specified --> Sized: power analysis
    Sized --> Frozen: fix items and grader
    Frozen --> Run: both models, same items
    Run --> Analysed: paired, cluster-robust
    Analysed --> Reported: difference + CI + n
    Analysed --> Frozen: add questions (declared)
    Reported --> [*]
    Frozen --> Sized: effect size changed? re-power

The transition that ruins everything is the one not drawn: Run --> Frozen with undeclared item additions after seeing the score. That is optional stopping, and it inflates the false-positive rate without leaving a trace in the report.

[IMAGE: Line chart of 95 percent confidence interval half-width against number of questions on a log x-axis from 50 to 100,000, at 70 percent accuracy, with vertical annotations marking AIME (15), GPQA Diamond (198), SWE-bench Verified (500), and MMLU (15,908). Caption: "Every commonly cited hard benchmark sits on the steep part of the curve."]

Watch It Run

By the Numbers

Sources: GPQA size and expert accuracy of 65 percent, rising to 74 percent after discounting mistakes experts identified in retrospect (Rein et al., 2023, arXiv:2311.12022); MMLU size of 15,908 questions over 57 subjects (Hendrycks et al., 2020, arXiv:2009.03300); MMLU error rates from the MMLU-Redux re-annotation of 5,700 questions across all 57 subjects (Gema et al., 2024, arXiv:2406.04127); SWE-bench sizes of 2,294 instances and a 500-instance human-validated subset (Jimenez et al., ICLR 2024, arXiv:2310.06770; OpenAI, 2024). Standard errors are computed from the binomial formula at 70 percent accuracy and are naive, ignoring clustering; the true intervals for MMLU and SWE-bench are wider.

The Virology row is the one worth staring at. It is a subset small enough to have a ±7 point interval and defective enough that the majority of analysed items were wrong, and it contributes to an aggregate that gets quoted to one decimal place.

[IMAGE: Grouped bar chart comparing naive versus cluster-corrected 95 percent confidence intervals for MMLU at three intra-cluster correlation values (0.05, 0.15, 0.30), showing the interval roughly doubling at the highest. Caption: "The correction nobody applies, sized."]

A Concrete Example

Two models are evaluated on an internal 250-question benchmark assembled from 25 topics, 10 questions each. Model A scores 178 correct, Model B scores 174. The launch draft says A beats B by 1.6 points.

Step 1: the naive intervals.

\(\bar{x}_A = 178/250 = 0.712\), so \(\mathrm{SE}_A = \sqrt{0.712 \times 0.288 / 250} = \sqrt{0.000820} = 0.0286\), or 2.9 points.

\(\bar{x}_B = 174/250 = 0.696\), so \(\mathrm{SE}_B = \sqrt{0.696 \times 0.304 / 250} = \sqrt{0.000846} = 0.0291\), or 2.9 points.

Reported properly, that is \(71.2 \pm 5.6\) against \(69.6 \pm 5.7\). The intervals overlap almost entirely.

Step 2: the unpaired difference.

\(\mathrm{SE}(\bar{x}_A - \bar{x}_B) = \sqrt{0.0286^2 + 0.0291^2} = \sqrt{0.001666} = 0.0408\), or 4.1 points. The 95 percent interval on the 1.6-point gap is \(\pm 8.0\) points. The data are consistent with B being 6.4 points better.

Step 3: pair it.

Pull the per-question results. The models agree on 220 of the 250 questions. On the 30 where they disagree, A is right and B wrong on 17, B is right and A wrong on 13. Note that \((17 - 13)/250 = 0.016\), which reproduces the 1.6-point gap exactly, as it must.

The paired interval is \(1.6 \pm 4.3\) points, against \(\pm 8.0\) unpaired. Pairing nearly halved the interval for free. McNemar's test on the discordant pairs gives \(\chi^2 = (|17-13| - 1)^2 / 30 = 9/30 = 0.30\), so \(p \approx 0.58\). Still nowhere near significant, but now the conclusion rests on the right calculation.

Step 4: correct for clusters.

The 250 questions came in 25 topic blocks of 10. Estimating the intra-cluster correlation of the differences at \(\rho = 0.15\) gives \(\mathrm{DEFF} = 1 + 9(0.15) = 2.35\), so the corrected standard error is \(0.0219 \times \sqrt{2.35} = 0.0336\). The honest interval is \(1.6 \pm 6.6\) points.

Step 5: what would have been needed.

For a paired binary comparison, the required \(n\) to detect a true effect \(\delta\) at \(\alpha = 0.05\) and 80 percent power is approximately

Holding the observed disagreement structure and targeting a true 2-point gap: \(n \approx 7.85 \times 0.1197 / 0.0004 \approx 2{,}350\) questions, before any clustering correction. With \(\mathrm{DEFF} = 2.35\), it is roughly 5,500.

The unpaired design needs more still: \(n \approx 7.85 \times (0.712 \times 0.288 + 0.696 \times 0.304) / 0.0004 \approx 8{,}200\) per model. That ratio, roughly 3.5 to 1, is what pairing is worth in questions you do not have to write.

[IMAGE: Forest plot of the same comparison analysed four ways, one row each: unpaired difference (1.6 ± 8.0), paired (1.6 ± 4.3), paired plus cluster correction (1.6 ± 6.6), and the target effect size of 2.0 marked as a vertical reference line, with zero marked as a dashed line every interval crosses. Caption: "Four analyses of one dataset. Only the width changes, and only one width is correct."]

The finding. A is not better than B on this benchmark. The benchmark is 250 questions and the difference being claimed is 1.6 points; those two facts are incompatible. The correct action is to report a tie, and either raise the effect size that matters or build a much bigger eval.

Where It Breaks

Statistical significance is not practical significance

At 20,000 questions a 0.3-point gap becomes significant and stays irrelevant. Significance answers "is the difference non-zero", which is nearly always yes given enough data, and never answers "is the difference worth switching models for". Decide the minimum effect size of interest before running, size the eval to detect it, and report the confidence interval rather than only the p-value so a reader can see whether practically meaningful values are inside it.

Systematic error does not shrink with \(n\)

Everything above concerns sampling error. Label errors are a different animal: they bias every model's score in a way that more questions from the same flawed generator do not fix. MMLU-Redux found an estimated 6.49 percent of MMLU items contain ground-truth errors, and 57 percent of the analysed Virology items, with model rankings shifting materially once the errors were corrected (Gema et al., 2024, arXiv:2406.04127).

Two models can also be affected differently by the same broken item. A model that reproduces the benchmark's mistaken answer key scores higher than a model that is right, and the direction of that bias correlates with what each model was trained on.

Multiple comparisons

Report 15 benchmarks and highlight the three wins, and the probability of at least one false positive at \(\alpha = 0.05\) under the null is \(1 - 0.95^{15} \approx 0.54\). Nobody has to cheat for this to happen. The mitigations are ordinary: pre-register the headline benchmarks, correct the threshold when scanning many, and report all of them rather than the selection.

The super-population may not exist

The framework assumes the eval items are exchangeable draws from a population you care about. For a hand-curated adversarial set built to expose a specific failure, they are not. A confidence interval computed over that sample answers "what would I score on more items drawn the same way", which is a question nobody asked, because the set was deliberately not a random sample of anything.

Optional stopping and the leaderboard version of it

Running the eval, adding questions, running again, and stopping when the result looks right inflates the false-positive rate substantially, and it leaves no trace in a report that shows only the final \(n\).

The commercial version of the same statistical sin is worse because it is invisible from the outside. If a provider can test many private model variants and choose which score to disclose, the published number is a maximum over draws rather than a draw, which biases it upward even when every individual measurement is unbiased. An audit of Chatbot Arena documented exactly this pattern, including 27 private variants tested by a single provider ahead of one release, alongside sampling-rate and deprecation asymmetries between proprietary and open models (Singh et al., 2025, The Leaderboard Illusion, arXiv:2504.20879).

Judges have their own error rate

When the grader is an LLM rather than a string match or a test suite, its disagreement with human labels enters the variance and, worse, the bias. A judge with 85 percent human agreement cannot adjudicate a 3-point difference between models, and if its errors correlate with a stylistic property such as response length, the bias does not average out with more items.

[IMAGE: Scatter plot of published headline gap between consecutive frontier model releases on GPQA Diamond, against the 95 percent confidence interval half-width for n=198, with a shaded band showing the region where gaps are indistinguishable from zero. Caption: "Most reported generational gaps on small hard benchmarks land inside the band."]

Alternative Designs

The first row is the status quo and the second row is the fix. Almost every team can move from row one to row two this afternoon, because the only requirement is logging per-item results rather than a total.

The last row is where the field is heading and is worth naming. Item response theory treats each question as having its own difficulty and discrimination parameters, which lets you identify items that no longer separate any model (dead weight) and items that separate strongly (the ones worth writing more of). It is the standard tool in educational testing, and it is the natural answer to "which 200 of my 2,000 questions are actually doing work".

How It Is Used in Practice

Frontier labs have converged on parts of this without always publishing the machinery. Reporting mean-of-\(K\) scores rather than single runs on small reasoning benchmarks is now common, which is the resampling recommendation in disguise. Model cards increasingly show a spread or a number of runs alongside the headline.

Where it is most valuable is not the launch post but the internal loop. A team fine-tuning a model runs dozens of comparisons a week, each between checkpoints that differ by fractions of a point. Without paired analysis and a declared effect size, that loop is a random walk that feels like progress: every iteration produces a number, half of the numbers go up, and the team ships whichever checkpoint was lucky on the day. Adding paired intervals to the internal eval harness is one of the highest-return pieces of infrastructure a modelling team can build, and it costs an afternoon.

The other practical application is knowing when to stop. If the power analysis says you need 5,500 questions to resolve the difference you care about and you have 300, the honest output is not a worse-powered test. It is a decision to either accept both models as equivalent on this axis and choose on cost or latency, or to invest in a bigger eval before investing further in modelling.

[IMAGE: Two-panel figure. Left: a sequence of 20 weekly checkpoint comparisons plotted as point estimates only, showing an apparent upward trend. Right: the same 20 comparisons with paired 95 percent intervals, showing every interval crossing zero. Caption: "The same data, before and after error bars."]

Insights Worth Remembering

1. A benchmark score is an estimate, and it has a standard error whether or not you print one. The number does not become certain because it was reported to one decimal place. At 250 questions and 70 percent accuracy, that error is 2.9 points, which is larger than most differences anyone publishes.

2. Pairing is free and buys about a third of the variance. It requires only that you log per-item results and that both models see identical items. No other single change to an eval report improves its resolution as much for as little.

3. Overlapping error bars do not mean no difference. The eyeball test on two independent intervals is conservative in a way that reads as rigour and is actually a different error. Compute the interval on the difference.

4. Clustering is the largest uncorrected error in the field. Subject blocks, repository groupings and shared passages all violate independence, and the resulting standard-error inflation can reach a factor of three. Most published intervals on MMLU-style benchmarks are too narrow.

5. More samples per question hit diminishing returns fast; more questions do not. Averaging \(K\) generations shrinks only the within-question term, which is usually the smaller one. Past roughly \(K = 5\), spend on items.

6. Lowering temperature is not variance reduction. It changes what you are measuring. A tight interval around the wrong quantity is worse than a wide interval around the right one, because it is more persuasive.

7. Systematic error is immune to sample size. Label errors, ambiguous items and a mistaken answer key bias every model's score, and adding more items from the same generator adds more of the same bias. Auditing a sample of your items by hand is a different and equally necessary activity from computing an interval.

8. Selection over repeated private measurements produces an upward-biased public number. This applies to leaderboards where providers pre-test variants, and identically to an internal team that runs the eval until it likes the result. The statistic is a maximum, not a draw.

9. Decide the effect size before you look at the data. Power analysis run afterwards is a rationalisation. Run beforehand, it frequently tells you the comparison you were about to make is impossible with the eval you have, which saves the work of making it.

Open Questions

How large is the intra-cluster correlation on real benchmarks? The correction is well understood and the inputs are not. Almost nobody publishes per-subject or per-repository item-level results, so \(\rho\) is usually guessed. Measuring and publishing it for MMLU, SWE-bench and the major reasoning suites would let every downstream report be corrected properly, and it requires no new methodology, only released item-level data.

Can item response theory make small benchmarks behave like large ones? IRT extracts more information per item by modelling difficulty and discrimination, and work such as tinyBenchmarks has shown that carefully chosen small subsets can approximate full-benchmark rankings (Polo et al., 2024, arXiv:2402.14992). Whether that survives on the hard, small, high-stakes benchmarks where it would matter most is untested, because those benchmarks do not have enough items for the fit.

What is the right significance framework when the grader is a model? Judge error is neither independent of the item nor of the model being judged, so it violates the assumptions of the standard corrections. Some work reports judge-human agreement and treats it as a fixed discount, which is almost certainly too simple. There is no accepted method here yet.

Does per-question difficulty transfer across models? Pairing works because question difficulty is a shared component. It is an open empirical question how shared it really is between models with very different training data, and if the answer is "less than assumed", the variance reduction from pairing is smaller than reported for cross-family comparisons.

Would pre-registration work in this field? It is standard in clinical trials and unheard of in model launches. The obvious objection is commercial: a lab that pre-registers loses the option to report the benchmarks it happened to win. Whether a norm could emerge for academic and open-model releases, where the incentive is weaker, is a governance question rather than a statistical one.

Sources and Further Reading

1. Miller, E. (2024). "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations." arXiv:2411.00640
2. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2023). "GPQA: A Graduate-Level Google-Proof Q&A Benchmark." arXiv:2311.12022
3. Gema, A. P., et al. (2024). "Are We Done with MMLU?" NAACL 2025. arXiv:2406.04127
4. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020). "Measuring Massive Multitask Language Understanding." arXiv:2009.03300
5. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023). "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024. arXiv:2310.06770
6. OpenAI (2024). "Introducing SWE-bench Verified." openai.com
7. Singh, S., et al. (2025). "The Leaderboard Illusion." arXiv:2504.20879
8. Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., et al. (2024). "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference." ICML 2024. arXiv:2403.04132
9. Liang, P., et al. (2022). "Holistic Evaluation of Language Models." arXiv:2211.09110
10. Srivastava, A., et al. (2022). "Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models." arXiv:2206.04615
11. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding." arXiv:1804.07461
12. Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024). "tinyBenchmarks: Evaluating LLMs with Fewer Examples." arXiv:2402.14992
13. Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv:2406.12045

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.