SWE-bench Explained: What the Percentage Actually Measures

SWE-bench is the most quoted coding benchmark in the field, and most of the quoting is careless. Numbers from different variants get compared as though they were the same measurement, which produces confident conclusions that are simply wrong.

It is worth understanding properly, because it is one of the better benchmarks available — and because knowing what it does not measure is as useful as knowing what it does.

What the benchmark does

Each task is a real issue from a real open-source repository, paired with the real commit that resolved it. The model receives the repository at the state before the fix and the text of the issue. It must produce a patch.

The patch is then applied and the repository's own test suite runs. If the tests that the real fix made pass now pass, the task is resolved. If not, it fails.

Two properties make this stronger than most benchmarks. The tasks are real rather than constructed, so they carry the mess of actual codebases. And scoring is mechanical rather than judged — no model or human decides whether the answer was good, the test suite does.

That second property matters more than it sounds. Judged benchmarks are vulnerable to verbosity bias, where longer and more confident answers score better regardless of correctness. A test suite is immune to persuasion.

The variants are not interchangeable

This is the mistake worth eliminating first.

SWE-bench is the original full set. SWE-bench Verified is a human-validated subset where the tasks were checked to be solvable and the tests to be fair — it removes items where the issue text was insufficient or the tests were broken. SWE-bench Pro is a harder set drawn to be more demanding.

Scores on these are not comparable. Qwen 3.6 27B reporting 77.2 percent on Verified and GLM-5.2 reporting 62.1 percent on Pro does not mean Qwen is ahead. They are different exams.

Whenever you see a SWE-bench figure, find the variant before forming an opinion. If the variant is not stated, treat the number as unusable rather than assuming.

What a percentage does not tell you

Resolution rate is one dimension, and several things that matter operationally are invisible in it.

It does not say how many attempts were allowed. A model given five tries and scored on any success is measured differently from one given a single shot. Harnesses vary here, and the difference is large.

It does not say how much the model spent. Two models resolving the same fraction can differ several-fold in tokens consumed, and cost per resolved issue is frequently the number that decides a real deployment.

It does not say whether the patch was good. The tests pass; that is all. A patch that passes tests while duplicating logic, ignoring conventions, or introducing a subtle inefficiency scores identically to a clean one.

The distribution is narrower than it appears

Tasks are drawn largely from popular Python open-source repositories with strong test coverage — the qualifying criterion, since the benchmark needs tests to score against.

Your codebase probably differs. Different language, thinner tests, more internal convention, less public discussion of its patterns in any training corpus. A model that performs well on well-documented Python libraries is not guaranteed to perform equivalently on a private TypeScript service with sparse tests.

This is not a flaw in the benchmark — every benchmark samples something. It is a limit on transfer, and it argues for treating SWE-bench as a shortlist filter rather than a decision.

What it genuinely predicts

Despite the caveats, a strong SWE-bench Verified result is meaningful. It means the model can read an unfamiliar codebase, locate the relevant region from a natural-language description, and produce a change that satisfies tests written by someone else.

That is a real and non-trivial capability, and it correlates with usefulness on bounded engineering tasks — bug fixes, small features, well-specified changes.

What it correlates with less is long unattended agentic work. Those tasks are bounded, and bounded benchmarks do not capture the compounding failure that makes forty-step runs hard. For that, Terminal-Bench 2.1 and FrontierSWE are the better signals. Why benchmarks disagree covers the distinction.

How to use it

Use SWE-bench to build a shortlist, never to make the final call. A model well below the field on Verified is unlikely to surprise you on your own code, so it is a reasonable filter.

Then run your own evaluation. Take fifteen closed issues from your own repository where you have the real fix, feed the issue text to each candidate, and check the patch against your own tests. That is SWE-bench methodology applied to your distribution, and it is the only version whose ranking you should trust for your decision.

How to benchmark LLMs on your own work covers building it, and the setup cost is one afternoon that pays back on every subsequent model release.