Reasoning & Evaluation
24 min
Your Test Set Is Wrong: Label Errors, Annotator Disagreement and the Ceiling on Measured Accuracy
Human reviewers confirmed 2,916 label errors in the ImageNet validation set, and an expert audit suggests the true figure is closer to one image in five. Every benchmark score is computed against an answer key written by people w…