SWE-bench and Agentic Coding Evaluation
What resolving a real GitHub issue measures that a function-completion benchmark does not, the contamination and scoping problems that complicate it, and how to read a reported score.
Function-level benchmarks ask a model to write a self-contained function from a docstring, with tests checking correctness. They measured something real and saturated, and they never resembled software engineering, which is mostly about changing existing code under constraints you did not choose.
SWE-bench (Jimenez et al., 2024, ICLR) changed the shape of the task: given a real repository at a real commit and a real issue report, produce a patch that makes the repository's own tests pass.
What it requires
The task cannot be done by generation alone. The system must locate the relevant code in a repository too large to read, understand existing behaviour, write a change consistent with surrounding conventions, and produce a patch that applies. Evaluation is by executing the maintainers' actual tests, including tests written after the issue, so it is a genuine functional check rather than a similarity score.
This is why performance on it tracks agentic capability rather than code generation: the difference between systems is largely in navigation, tool use and iteration rather than in the quality of any single generated block.
How to read a score
Contamination is the first question. The issues and their fixes are public and predate most models' training cutoffs, so a model may have seen the resolution. SWE-bench Verified, a human-validated subset, and later variants constructed from post-cutoff repositories exist to address this, and a score should always be attributed to a specific variant.
Scaffolding is doing much of the work. The same model scores very differently under different agent harnesses, so a reported number describes a model-plus-scaffold pair. Comparing models across papers that used different scaffolds compares two systems on two axes at once.
Test-passing is not correctness. A patch can pass the tests while being wrong in ways the tests do not cover, and models can produce changes that satisfy tests by narrowing behaviour. Manual inspection of passing patches finds this at a non-trivial rate.
The distribution is narrow. The benchmark is Python, drawn from a set of popular libraries, and the issues are ones that a patch can resolve. Performance on it says little about a different language, an unfamiliar codebase, or a task requiring design rather than repair.
When it breaks
Benchmarks drive optimisation toward themselves. A public benchmark with visible tests invites scaffolds tuned to its structure, and improvements can be specific to the harness rather than general. Held-out and continuously refreshed variants are the response.
Cost is rarely reported and varies enormously. An agent making two hundred model calls to resolve an issue and one making twenty may score similarly, and they are not comparable systems. Score per dollar and score per wall-clock minute belong beside the resolution rate.
Passing rate hides catastrophic attempts. A system that resolves 40 percent and leaves the repository broken in another 10 percent is different from one that resolves 40 percent and cleanly fails otherwise. Failure quality is not captured by a resolution rate.
No benchmark covers review, design or maintenance. Reading a change for correctness, deciding what to build, and keeping a system working over years are the bulk of software engineering, and none of them appear.
12 flashcards for this concept
Click a card to reveal the answer.