Agentic Benchmarks: SWE-bench and Its Descendants

SWE-bench takes 2,294 real GitHub issues from 12 popular Python repositories, hands the model the repository at the parent commit plus the issue text, and asks for a patch. Grading runs the project's own tests: the fail-to-pass tests from the merged pull request must now pass, and the pass-to-pass tests must not regress (Jimenez et al., ICLR 2024, arXiv:2310.06770).

Nothing about that scoring involves a judge model or a string comparison. That is the design's central virtue, and it is what separates agentic benchmarks from the multiple-choice era.

Why execution-based grading changes the game

A multiple-choice benchmark has a fixed answer key, so it can be memorised, and the score is a single sample from a single forward pass. An execution-graded benchmark has a verifier, which changes three things at once.

The reward is not gameable by fluency. A patch either makes the test pass or it does not, and no amount of confident explanation substitutes. The task is multi-step, so it exercises navigation, tool use, and recovery rather than knowledge recall. And the same verifier that grades can be used as a training signal, which is why this benchmark family and RLVR grew up together.

The cost is that the environment is now part of the benchmark. Docker images, dependency pinning, flaky tests and network access all become confounds.

SWE-bench Verified, and what the correction revealed

The original set had a quality problem. Some issues were unsolvable from the information given, some test patches checked behaviour the issue never described, and some tasks required knowing the repo's future. OpenAI worked with the SWE-bench authors to have annotators review instances and released SWE-bench Verified, a 500-instance human-validated subset (OpenAI, 2024, Introducing SWE-bench Verified).

Scores on Verified are substantially higher than on the full set for the same systems, and the difference is almost entirely benchmark defect rather than capability. This is worth internalising as a general result: on a hard, real-world benchmark, a meaningful fraction of "model failures" are task defects, and nobody finds them until someone reads every instance.

The harness confound

A SWE-bench number describes a system, not a model. That system includes the retrieval strategy for finding relevant files, the tool set, the number of attempts allowed, whether the agent may run tests before submitting, and the scaffold's error-recovery logic. The same underlying model varies by tens of points across published scaffolds.

This makes cross-paper comparison close to meaningless unless the harness is held fixed. The practical response is to report the scaffold explicitly, and to compare a new model against a baseline model in your own harness rather than against a leaderboard row.

The rest of the family

τ-bench is the one to study for the metric rather than the domain. Its pass^k requires the same task to succeed on all \(k\) independent runs, which exposes the policy inconsistency that a single-run pass rate hides (Yao et al., 2024, arXiv:2406.12045).

When it breaks

Contamination is worse here, not better. The repositories are public, the issues are public, and the merged fix is public. A model trained after the issue was resolved may have seen the patch. Date-filtered variants exist for this reason and need to be used deliberately.

Success rate hides cost. Two agents at 45 percent are not equivalent if one spends 30,000 tokens per instance and the other spends 900,000. Report tokens and wall-clock alongside the score, or the benchmark rewards brute force.

Passing the tests is not fixing the issue. The verifier checks the tests that existed. An agent that special-cases the test input passes; reward hacking against unit tests is documented and common, and the only defence is holding out tests the agent never sees.