Your Test Set Is Wrong: Label Errors, Annotator Disagreement and the Ceiling on Measured Accuracy

In 2021, Curtis Northcutt, Anish Athalye and Jonas Mueller sent 5,440 suspicious ImageNet validation labels to crowd workers, five per image, and asked whether each label matched the picture. For 2,916 the answer was no (Northcutt, Athalye & Mueller, 2021, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, arXiv:2103.14749). That is 5.83% of the set the field used for a decade to rank vision models, and it counted only the errors their algorithm thought to look for. When experts checked a sample of images the algorithm had not flagged, 16% had problems too, and the authors revised the whole-set estimate to "closer to 20%".

Every accuracy number compares a model with an answer key written by people who disagree, misread guidelines and face items with no single right answer. Label error caps measured accuracy, compresses the gaps between models and, under common conditions, reverses the ranking. The same arithmetic governs LLM benchmarks, where the key is a multiple-choice letter or a unit test.

Why this matters: A benchmark score is only as trustworthy as its labels. Estimating label error from annotator disagreement, knowing how that error bounds and distorts accuracy, and telling real errors from legitimate disagreement is what separates a meaningful leaderboard gain from a model that has learned the annotators' mistakes.

TL;DR

• Across ten standard test sets, human-validated label errors average at least 3.3%, from 0.15% (MNIST) to 10.12% (QuickDraw); ImageNet validation has at least 5.83%, and an expert audit puts it nearer 20% once multi-label images count.
• With test-label error rate \(\varepsilon\), a perfect model measures at most \(1-\varepsilon\), and on binary tasks every true gap between models shrinks by a factor of \(1-2\varepsilon\).
• A model that learned the annotators' systematic mistakes can score above a better model. On corrected ImageNet labels, ResNet-18 overtakes ResNet-50 if mislabeled prevalence rises by about 6 points.
• Two annotators who disagree on a fraction \(d\) of items have an average error of at least \(d/2\), with no independence assumption. It is the cheapest ceiling estimate.
• Kappa and alpha measure reliability, not correctness: a rare-class task can show 98% agreement and \(\kappa = 0.32\).
• LLM benchmarks have the same disease: annotators filtered out 68.3% of 1,699 sampled SWE-bench tasks, and MMLU-Redux estimates 6.49% of MMLU questions are flawed.
• Some disagreement is not error: on ChaosNLI, models are near chance where 100 annotators disagree.

At a Glance

flowchart LR
    T["True label, unknown"] --> H["Annotators apply a guideline"]
    H --> G["Given test label"]
    H -.->|"disagreement rate d"| E["Error at least d/2"]
    M["Model prediction"] --> S["Measured accuracy"]
    G --> S
    E --> C["Ceiling: 1 minus epsilon"]
    C --> S
    G -.->|"systematic errors learned"| F["Ranking can flip"]
    F --> S

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class T,H blue
    class M purple
    class G slate
    class S teal
    class E,C amber
    class F rose

Measured accuracy depends on the model and the labelling process; disagreement is the observable trace of that process.

Before Anyone Checked the Answer Key

Measuring agreement between judges predates machine learning. Jacob Cohen introduced kappa in 1960 for clinicians whose raw agreement was inflated by whichever diagnosis was common (Cohen, 1960, Educational and Psychological Measurement 20(1)). Dawid and Skene then treated each rater as a noisy channel with its own confusion matrix and recovered error rates and latent true labels with EM (Dawid & Skene, 1979, Applied Statistics 28(1)); serious relabelling projects still aggregate raters this way.

Machine learning inherited the datasets but not the discipline: one label per item, the test split ground truth by definition, and a label-noise literature concerned almost entirely with training labels.

timeline
    title From clinician agreement to benchmark audits
    1960 : Cohen proposes kappa for two raters
    1979 : Dawid and Skene estimate per-rater error rates with EM
    1990 : Feinstein and Cicchetti publish the two kappa paradoxes
    2015 : Aroyo and Welty publish seven myths of human annotation
    2019 : Recht et al. rebuild ImageNet and see 11 to 14 point drops
         : CIFAR-10H collects about 51 labels per test image
    2020 : Beyer et al. relabel ImageNet validation as multi-label ReaL
         : Engstrom et al. trace most of the ImageNetV2 drop to selection bias
    2021 : Northcutt et al. find at least 3.3 percent test errors in ten sets
    2024 : SWE-bench Verified keeps 500 human-screened tasks
         : MMLU-Redux estimates 6.49 percent of MMLU is flawed

The shift came sideways. Rebuilding CIFAR-10 and ImageNet test sets by the original procedures, they found drops of 3 to 15 points on CIFAR-10 and 11 to 14 on ImageNet, "approximately five years of progress", and blamed slightly harder images rather than adaptivity (Recht et al., 2019, ICML, arXiv:1902.10811). Engstrom and colleagues then showed that matching the new set on a noisy crowd "selection frequency" biases the sample; after correction only 3.6 ± 1.5 of an 11.7-point drop remained unexplained (Engstrom et al., 2020, ICML, arXiv:2005.09619). The lesson: a human labelling process between data and metric can move the metric more than the models do.

[IMAGE: Two-panel scatter plot. Left: original ImageNet accuracy versus ImageNetV2 accuracy for about 60 models, all 11 to 14 points below the diagonal on a straight line. Right: the same models after Engstrom et al.'s bias correction, gap shrunk to a few points. Caption: "Was the drop about the models or about the relabelling?"]

How Label Error Caps Measured Accuracy

Measured accuracy is a mixture

Let \(y^*\) be the true label, \(\tilde y\) the test label and \(\hat y\) the prediction. Everyone reports \(\tilde A = P(\hat y = \tilde y)\), which Northcutt et al. call original accuracy; everyone means \(A^* = P(\hat y = y^*)\), corrected accuracy. Let \(\varepsilon = P(\tilde y \ne y^*)\). Splitting on whether the label is right:

The second term is a wrong model agreeing with a wrong label, and the metric rewards it. If label errors are independent of model errors, the first term is \(A^*(1-\varepsilon)\). Let \(c\) be the probability that a wrong prediction coincides with a wrong label when both are wrong:

A perfect model scores \(1-\varepsilon\): the ceiling. With \(K\) classes and scattered errors, \(c \approx 1/(K-1)\), negligible on ImageNet's 1,000 classes. On a binary task \(c = 1\) and \(\tilde A = A^*(1-2\varepsilon) + \varepsilon\), a line with slope \(1-2\varepsilon\).

Gaps shrink, then flip

That slope is an attenuation factor: a true gap \(\Delta\) measures as \(\Delta(1-2\varepsilon)\), and what remains must still clear sampling error (see error bars for LLM evals). Attenuation preserves rankings. Correlated error does not. Label errors cluster on confusable classes and ambiguous guideline boundaries, and a model trained on labels from the same process learns the same confusions. On exactly the mislabeled items it agrees with the wrong label far more than chance: a large \(c\) where \(\varepsilon\) lives, and a score above its true accuracy.

Northcutt et al. observed this directly. Higher-capacity models "undesirably reflect the distribution of systematic label errors in their predictions to a greater degree than models with fewer parameters". On their pruned ImageNet set, where correctable errors are 2.9% of items, ResNet-50 leads; on corrected labels the two cross when the mislabeled share reaches about 9%, so "ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled test examples increases by just 6%". On CIFAR-10, VGG-11 overtakes VGG-19 at a 5-point increase. The better deployment choice, they conclude, is "unknowable without correcting the test data labels".

[IMAGE: Line chart, noise prevalence 3 to 50 percent on x, corrected top-1 accuracy on y, for ResNet-18, ResNet-50 and NASNet, with dotted verticals where pairs cross, the first near 9 percent. Caption: "Bigger models degrade faster on corrected labels because they learned the original annotators' mistakes."]

From disagreement to an error rate

The ceiling needs \(\varepsilon\), which is never observed. Disagreement is, if items are double-labelled. If there is one true label and annotators A and B disagree, at least one is wrong:

So the average error is at least \(d/2\), with no independence or symmetry assumption. Assume independent errors at a common rate \(e\) on a binary task and the floor becomes an estimate:

Distrust the independence. Shared guideline ambiguity makes annotators wrong together, which lowers disagreement and hides error. Disagreement measures the idiosyncratic part of annotation error and misses the systematic part, the part that flips rankings. The same arithmetic prices adjudication: majority-of-three with independent errors is wrong with probability \(3e^2 - 2e^3\), but shared errors survive any number of votes from the same pool.

Kappa, alpha and their paradoxes

Cohen's kappa subtracts the agreement two annotators would reach labelling at random with their own class rates:

with \(p_o\) the observed agreement and \(p_{A,k}\) the share annotator A put in class \(k\). Krippendorff's alpha generalises to many coders, missing data and ordinal or interval scales, as \(\alpha = 1 - D_o/D_e\) with expected disagreement \(D_e\) computed from the pooled label distribution (Hayes & Krippendorff, 2007, Communication Methods and Measures 1(1)). Krippendorff's conventional thresholds: rely on \(\alpha \ge 0.800\), draw only tentative conclusions from 0.667 to 0.800, discard below (Krippendorff, 2004, Content Analysis, pp. 241-243).

Both measure reliability; the ceiling needs error. Feinstein and Cicchetti documented why the two diverge (Feinstein & Cicchetti, 1990, Journal of Clinical Epidemiology 43(6)). In the prevalence paradox, a rare class drives \(p_e\) toward \(p_o\): two screeners of 1,000 items who both flag 5, split on 20 and both clear 975 agree 98% of the time, but each flags 1.5%, so \(p_e = 0.015^2 + 0.985^2 = 0.970\) and \(\kappa = 0.32\), while the disagreement bound puts error at a mere 1% or more. In the bias paradox, annotators with different class rates lower \(p_e\) and raise kappa, so disagreement about prevalence flatters reliability. Gwet's AC1 was built to resist the first (Gwet, 2008, British Journal of Mathematical and Statistical Psychology 61(1)); the safer habit is to report prevalence, bias and the raw confusion table next to any kappa.

Finding the errors: confident learning

Disagreement says how many errors; confident learning estimates which (Northcutt, Jiang & Chuang, 2021, JAIR 70, arXiv:1911.00068). From out-of-sample predicted probabilities it sets, for each class \(j\), a threshold equal to the model's mean self-confidence on items labelled \(j\):

An item labelled \(i\) counts as probably truly \(j\) if \(\hat p(\tilde y = j; x) \ge t_j\) (arg-max among classes that clear their thresholds). The counts form the confident joint \(C_{\tilde y, y^*}\), whose off-diagonal cells are estimated errors; calibrating rows to observed class counts and normalising gives \(\hat Q_{\tilde y, y^*}\), and one minus its trace is the error rate. Per-class thresholds matter: a plain arg-max over-assigns items to classes the model is overconfident about. On ImageNet's training set the largest off-diagonal cells were ontology problems rather than slips: 645 "projectile" images judged "missile", a duplicated "maillot" class, "corn" versus "ear".

It ranks rather than rules: only 51% of flagged test items were confirmed errors, but a flagged ImageNet image was 2.6 times as likely to be mislabeled as an unflagged one.

When there is no single truth

All of this assumes \(y^*\) exists. Aroyo and Welty's seven myths of human annotation begin with one truth and disagreement is bad, arguing that for interpretive tasks the spread of judgements is signal about the item (Aroyo & Welty, 2015, AI Magazine 36(1)). Plank calls it human label variation and argues it must be modelled through data, training and evaluation rather than aggregated away (Plank, 2022, EMNLP, arXiv:2211.02570).

The evidence is strong. CIFAR-10H gathered 511,400 judgements for the 10,000 CIFAR-10 test images, and training on the full distributions improved out-of-distribution generalisation (Peterson et al., 2019, ICCV, arXiv:1908.07086). ChaosNLI collected 100 labels for each of 4,645 NLI examples; models "achieve near-perfect accuracy on the subset of data with a high level of human agreement, whereas they can barely beat a random guess" where agreement is low, and those items account for most of their errors (Nie, Zhou & Bansal, 2020, EMNLP, arXiv:2010.03532).

So the ceiling has two parts. Error is a label competent annotators would reject on reflection; correct it. Variation is where they legitimately differ; keep the distribution or an answer set. Northcutt et al.'s 2,916 ImageNet errors split this way: 1,428 correctable to one class, 597 multi-label, 293 neither class, 598 with no majority.

Seeing It in Motion

A model nominates suspects and people rule on them: Northcutt et al.'s protocol, plus the expert audit that estimated what it missed.

sequenceDiagram
    participant M as Model, cross-validated
    participant CL as Confident learning
    participant W as Crowd, 5 per item
    participant X as Expert reviewers
    participant B as Maintainer
    M->>CL: out-of-sample probabilities
    CL->>W: 5,440 flagged ImageNet items
    W->>B: 2,916 errors at 3-of-5 threshold
    Note over W,B: correctable, multi-label, neither, no majority
    B->>X: one flagged and one unflagged image per class
    X->>X: two experts per image, agree on 77 percent
    X->>B: unflagged images still 16 percent problematic
    Note over B: estimate revised from 6 to about 20 percent

Before anyone "fixes" a disagreement, it should pass through a decision like this, because each branch needs a different action.

flowchart TB
    D["Annotators disagree"] --> Q1{"Would an expert reject one label?"}
    Q1 -->|"yes"| ERR["Label error: correct it"]
    Q1 -->|"no"| Q2{"Guideline covers the case?"}
    Q2 -->|"no"| GAP["Guideline gap: add a ruling"]
    Q2 -->|"yes"| Q3{"Several labels defensible?"}
    Q3 -->|"yes"| VAR["Human label variation"]
    Q3 -->|"no"| ERR
    VAR --> SOFT["Keep distribution or answer set"]
    GAP --> REL["Relabel affected batch"]
    ERR --> CEIL["Recompute epsilon and ceiling"]
    SOFT --> CEIL
    REL --> CEIL

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class D blue
    class Q1,Q2,Q3 purple
    class ERR rose
    class GAP amber
    class VAR,SOFT emerald
    class REL slate
    class CEIL teal

[IMAGE: Four ImageNet images, original label in red and relabelled answer in green: correctable ("white stork" to "black stork"), multi-label ("mantis" also "fence"), neither, no majority. Caption: "Only the first kind of error has one right answer."]

By the Numbers

The classic audit. "Estimated error" is Northcutt et al.'s lower bound: validated errors among flagged items over test-set size, so errors the algorithm never flagged are not counted.

Relabelling and re-screening, into the LLM era:

Sources: Northcutt et al. (2021), Tables 1-3 and Section 6; Beyer et al. (2020); Recht et al. (2019); Engstrom et al. (2020); OpenAI (2024); Gema et al. (2024); Yu et al. (2025). The 20% ImageNet figure is the authors' extrapolation from a two-per-class sample that counts multi-label images; treat it as an estimate. SWE-bench flags overlap, so the percentages do not sum.

[IMAGE: Horizontal bars of the ten error rates from MNIST (0.15%) to QuickDraw (10.12%), with a lighter extension on ImageNet to about 20% labelled "expert-audit estimate". Caption: "The CL figures are floors; the one set audited for what CL missed went from 6% to roughly 20%."]

A Concrete Example

Before labelling a 1,000-item toxicity test set, a team has annotators A and B independently label the same 200 items.

Step 1. Tabulate.

Step 2. Kappa. \(p_o = (30 + 146)/200 = 0.880\). A calls 20% toxic, B 22%, so \(p_e = 0.20 \times 0.22 + 0.80 \times 0.78 = 0.668\) and

Step 3. Alpha, as a check. Pooled, 84 of 400 labels are toxic: \(D_e = 2 \times 84 \times 316 / (400 \times 399) = 0.3326\) and \(D_o = 24/200 = 0.120\), so \(\alpha = 1 - 0.120/0.3326 = 0.639\), below 0.667: too unreliable for firm conclusions.

Step 4. Floor on error. \(d = 0.12\), so \(e_A + e_B \ge 0.12\): the average annotator is wrong on at least 6.0% of items, and a single-annotator test set caps a perfect model near 94%, assumption-free.

Step 5. Estimate under independence. \(2e(1-e) = 0.12\) gives \(e = (1 - \sqrt{0.76})/2 = 0.0641\), a ceiling of 93.6%. Kappa never entered; the ceiling depends on raw disagreement, which kappa deliberately discounts.

Step 6. Price adjudication. Majority-of-three gives \(3e^2 - 2e^3 = 0.0123 - 0.0005 = 0.0118\), a 98.8% ceiling for triple the labelling cost, and only for independent errors.

Step 7. Attenuation. Two classifiers are truly 92% and 90% accurate. On the single-annotator test set, binary so \(c = 1\):

The 2.0-point gap measures 1.74 (\(\times\, 0.872\)). With 1,000 items the standard error near 0.87 is \(\sqrt{0.87 \times 0.13/1000} \approx 0.011\), about ±2.1 points at 95%: the gap is already below resolution.

Step 8. The flip. Now let 64 of the 1,000 test labels be wrong in a systematic way. Model P (truly 92%) errs independently of the annotators: about 861 correct on the 936 clean items, and on the 64 bad ones it predicts the truth on about 59 and matches the wrong label on 5. Measured: \(861 + 5 = 866\), 86.6%. Model Q (truly 90%) was trained on labels from the same pool and learned their sarcasm confusion: on the 64 bad items it reproduces the wrong label on 30 and the truth on 34, so its 900 correct predictions include 866 clean ones. Measured: \(866 + 30 = 896\), 89.6%. The worse model wins by 3.0 points, and only relabelling those 64 items reveals it.

[IMAGE: Two bars of 1,000 cells for models P and Q, cells coloured correct-on-clean (teal), right-but-scored-wrong on mislabeled items (amber), wrong-but-scored-right by matching a wrong label (rose); Q's rose band is visibly larger. Caption: "Q scores higher because it agrees with the annotators' mistakes."]

Where It Breaks

The disagreement bound misses correlated error

Annotators sharing a guideline, pool and interface share mistakes, which look like agreement to every coefficient here. Crowd verification of flagged ImageNet items found 5.83% and could not see the problems experts later estimated in 16% of the other 89%. Independent disagreement estimates annotation variance; only a stronger, independent labeller estimates its bias.

The detector shares the model's blind spots

Confident learning and ReaL both let models decide what humans see. Beyer et al. built candidate labels from 6 of 19 models, keeping 97.1% recall against expert labels, and sent no image to raters when every model agreed with the original label, 25,111 of 50,000 (Beyer et al., 2020, Are we done with ImageNet?, arXiv:2006.07159). An error all models share is never shown to a person, and LLM-based verification inherits the circularity. A model-blind random sample checked by people, as the LLM-assisted annotation concept argues, is the only exit.

Corrections have their own error rate

Only 51% of CL flags were real errors, and the false alarms include unusual views, such as a cropped fragment of a sewing machine or an airplane seen from its cockpit. Northcutt et al. note that even 5-of-5 crowd agreement on a correction does not guarantee it is right. Their experts took 67 seconds per image against the crowd's 5 and still agreed on only 77% before discussion. A corrected set has a smaller \(\varepsilon\), not zero, and should publish it.

Pruning moves the distribution

Every audit deletes ambiguity. Northcutt et al. exclude no-majority and multi-label items from corrected accuracy; Beyer et al. discard 3,163 unlabelled images; SWE-bench Verified keeps 500 of 1,699. Contested items are disproportionately the hard and realistic ones, so the filtered benchmark has a higher ceiling and a different difficulty profile, and its scores are not comparable with the original's. Filtering also leaves errors inside the kept items: UTBoost found insufficient tests admitting 345 wrong patches, affecting 24.4% of SWE-bench Verified leaderboard entries and changing 11 rankings (Yu et al., 2025, UTBoost, ACL, arXiv:2506.09289).

Treating variation as error

Correcting an item where competent annotators genuinely differ installs one opinion as truth and penalises the other defensible answer. On such items accuracy is not capped; it is undefined.

Alternative Designs

These are stages more than rivals, and none can recover a ceiling that was never measured; that needs overlapping independent labels planned in advance.

How It Is Used in Practice

LLM benchmark maintainers now publish screened subsets. For SWE-bench Verified, 93 experienced Python developers rated sampled tasks for underspecified issues and for tests that "may unfairly mark valid solutions as incorrect", on a 0-3 severity scale where 2 or 3 meant discard. The 500 surviving tasks, in OpenAI's words, supersede "the original SWE-bench and SWE-bench Lite test sets"; GPT-4o resolved 33.2% of them (OpenAI, 2024, Introducing SWE-bench Verified). The telling number is 68.3%: two thirds of a heavily used benchmark had an answer-key problem severe enough to discard.

MMLU-Redux re-annotated 5,700 MMLU questions across all 57 subjects with an error taxonomy (bad question clarity, bad options clarity, no correct answer, multiple correct answers, wrong ground truth) and reported "significant discrepancies" with originally published model performance (Gema et al., 2024, Are We Done with MMLU?, NAACL 2025, arXiv:2406.04127). Vendrow and colleagues' "platinum" revisions of fifteen benchmarks show frontier models still failing simple items once the key is trustworthy (Vendrow et al., 2025, arXiv:2502.03461).

For a team's own evaluation set: double-label a random 10 to 20% independently and compute \(d\); report the ceiling beside the score; adjudicate disagreements with a stronger labeller; keep test and training labelling processes separate so learned systematic errors are not rewarded; and run confident learning (the open-source cleanlab library) to prioritise re-review, spot-checking a random unflagged sample for what it missed.

[IMAGE: Evaluation scorecard: a 1,000-item test set with a 200-item double-labelled band, a disagreement meter at d = 12%, a ceiling of 93.6%, an adjudication queue, and a result line "model 86.6%, ceiling 93.6%, 95% CI ±2.1". Caption: "A score without its ceiling and error bar is half a measurement."]

Insights Worth Remembering

1. The ceiling belongs to the labels, not the model. When top entries bunch below a number that never moves, suspect the answer key before the architecture.

2. Disagreement is a free lower bound on error. Annotators disagreeing on \(d\) of items have average error of at least \(d/2\), with no assumptions. One overlapping batch buys it.

3. Kappa measures consistency, not correctness. It shows whether a guideline is applied uniformly, not how accurate the labels are.

4. Noisy keys compress gaps before they reverse them. Independent error scales true differences by \(1-2\varepsilon\); systematic error rewards learning the annotators' mistakes.

5. Automated audits find the errors models can see. The ImageNet estimate went from 6% to about 20% only when experts examined images the algorithm had passed.

6. A cleaned benchmark is a new benchmark. Filtering to unambiguous items raises the ceiling and shifts difficulty, so verified-subset scores do not continue the old series.

Open Questions

How close are frontier LLMs to their benchmark ceilings? MMLU-Redux measured 6.49% flawed questions; most other benchmarks have never been audited, so their ceilings are unknown.

Can systematic annotation error be estimated without a stronger labeller? Dawid-Skene identifies rater confusion under conditional independence, exactly what shared guidelines violate. Whether any label-only method can bound correlated error, rather than flag it, is open.

Should evaluation move to distributions? CIFAR-10H and ChaosNLI show soft labels are collectable and informative; which metric to report against them is unsettled.

Do LLM verifiers make audits better or more circular? Evidence that they find real errors is growing; evidence on which errors they systematically miss is thin.

What is \(\varepsilon\) for an executable answer key? In SWE-bench the label is a test suite, and an insufficient test is a false positive no annotator agreement reveals. UTBoost shows the effect is real; a principled way to estimate it is not yet established.

Sources and Further Reading

1. Northcutt, C. G., Athalye, A., & Mueller, J. (2021). "Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks." NeurIPS Datasets and Benchmarks. arXiv:2103.14749
2. Northcutt, C. G., Jiang, L., & Chuang, I. L. (2021). "Confident Learning: Estimating Uncertainty in Dataset Labels." JAIR, 70, 1373-1411. arXiv:1911.00068
3. Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., & van den Oord, A. (2020). "Are we done with ImageNet?" arXiv:2006.07159
4. Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019). "Do ImageNet Classifiers Generalize to ImageNet?" ICML. arXiv:1902.10811
5. Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Steinhardt, J., & Madry, A. (2020). "Identifying Statistical Bias in Dataset Replication." ICML. arXiv:2005.09619
6. Cohen, J. (1960). "A Coefficient of Agreement for Nominal Scales." Educational and Psychological Measurement, 20(1), 37-46. doi:10.1177/001316446002000104
7. Dawid, A. P., & Skene, A. M. (1979). "Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm." Applied Statistics, 28(1), 20-28. doi:10.2307/2346806
8. Feinstein, A. R., & Cicchetti, D. V. (1990). "High agreement but low kappa: I. The problems of two paradoxes." Journal of Clinical Epidemiology, 43(6), 543-549. doi:10.1016/0895-4356(90)90158-L
9. Hayes, A. F., & Krippendorff, K. (2007). "Answering the Call for a Standard Reliability Measure for Coding Data." Communication Methods and Measures, 1(1), 77-89. doi:10.1080/19312450709336664
10. Krippendorff, K. (2004). Content Analysis: An Introduction to Its Methodology (2^nd ed.). Sage.
11. Gwet, K. L. (2008). "Computing inter-rater reliability and its variance in the presence of high agreement." British Journal of Mathematical and Statistical Psychology, 61(1), 29-48. doi:10.1348/000711006X126600
12. Aroyo, L., & Welty, C. (2015). "Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation." AI Magazine, 36(1), 15-24. doi:10.1609/aimag.v36i1.2564
13. Plank, B. (2022). "The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation." EMNLP, 10671-10682. arXiv:2211.02570
14. Peterson, J. C., Battleday, R. M., Griffiths, T. L., & Russakovsky, O. (2019). "Human Uncertainty Makes Classification More Robust." ICCV, 9616-9625. arXiv:1908.07086
15. Nie, Y., Zhou, X., & Bansal, M. (2020). "What Can We Learn from Collective Human Opinions on Natural Language Inference Data?" EMNLP, 9131-9143. arXiv:2010.03532
16. OpenAI (2024). "Introducing SWE-bench Verified." openai.com
17. Gema, A. P., et al. (2025). "Are We Done with MMLU?" NAACL, 5069-5096. arXiv:2406.04127
18. Vendrow, J., Vendrow, E., Beery, S., & Madry, A. (2025). "Do Large Language Model Benchmarks Test Reliability?" arXiv:2502.03461; Yu, B., et al. (2025). "UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench." ACL. arXiv:2506.09289

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.