SWE-bench Leaders Compared: Reading the Scores Correctly

SWE-bench is the closest thing the field has to a shared definition of "can this model do software engineering". It is also the benchmark most often quoted wrongly, because it is not one benchmark. It is a family, and the variants are not interchangeable.

The most common error in a model comparison is putting a SWE-bench Verified score next to a SWE-bench Pro score in the same column. That comparison is meaningless, and it appears constantly.

What the benchmark actually asks

SWE-bench takes real issues from real open-source repositories, gives the model the repository state before the fix, and asks it to produce a patch. The patch is then run against the project's own tests. There is no partial credit and no human judgement — the tests pass or they do not.

That construction is what makes it valuable. Unlike a multiple-choice evaluation, it is very hard to score well by pattern-matching. The model has to locate the relevant code in a codebase it has not been given a map of, understand the intent of the issue, and make a change that does not break anything else.

It is also what makes it narrow. Every task has a clean issue description and a test suite that defines success exactly. Real work frequently has neither. The full explanation of SWE-bench covers the construction in more detail.

Verified against Pro

Verified is a human-filtered subset, built to remove tasks that were unsolvable, ambiguous or broken. It is the more forgiving variant and the one most numbers are quoted from.

Pro is the harder variant, built to resist the saturation that Verified was starting to show. The tasks are more involved and the ceiling is lower. A model scoring in the low sixties on Pro is not worse than a model scoring in the high seventies on Verified — you cannot tell from those two numbers which is stronger at all.

So the first discipline in reading any comparison is to check which variant each figure came from, and refuse to rank across them. If a table does not say, treat the table as unreliable rather than guessing.

The 2026 field, by variant

On SWE-bench Verified, DeepSeek V4 Pro reports 80.6 percent. Qwen 3.6 27B — a dense model that fits on a single accelerator — reports 77.2 percent, which is the more interesting of the two numbers because of what it costs to run.

On SWE-bench Pro, GLM-5.2 reports 62.1 percent. MiniMax M3 reports 59.0 percent, and that one carries a vendor-claimed label, meaning it has not been independently reproduced at the time of writing. Vendor-claimed figures are not automatically wrong, but they warrant more caution than a third-party evaluation.

Kimi K3 reports 81.2 on FrontierSWE, a different evaluation again in the same broad family. Same caveat applies: it does not slot into either of the columns above.

What a three-point gap is worth

The gap between DeepSeek V4 Pro at 80.6 and Qwen 3.6 27B at 77.2 on Verified is 3.4 points. It is tempting to read that as a small difference and move on, but it is worth being precise about what it means.

Benchmark differences of a few points are frequently within the noise introduced by the scaffold — the prompt, the retrieval strategy, the number of attempts allowed, whether the harness lets the model run tests before submitting. Two evaluations of the same model with different scaffolds can differ by more than the gap between two models.

That matters most when the two models differ enormously in deployment cost. Qwen 3.6 27B is dense and runs on one GPU; DeepSeek V4 Pro is a 1.6T mixture-of-experts activating roughly 49B per token. If three points is scaffold noise, the operational difference is the only real difference. What fits on one GPU works through that trade.

Where the benchmark stops predicting

SWE-bench measures a specific shape of work: a well-described bug in a public Python-heavy repository, with tests that already define correctness.

Your work probably differs on several axes at once. The issue description may be a Slack message. The codebase may be private, larger, and in a language underrepresented in the benchmark. There may be no test that captures the requirement. And a large share of real engineering is not bug-fixing at all — it is new features, refactors and migrations, none of which the benchmark touches.

The scores are a reasonable prior for ranking models. They are a poor substitute for evaluating on your own repository, which is not hard to do and is far more predictive. Benchmarking on your own work covers a lightweight version of that.

How to use these numbers

Treat SWE-bench as one input among several. Pair it with an agentic measure like Terminal-Bench, which tests tool use and error recovery rather than patch correctness, and with your own evaluation on real tasks. The Terminal-Bench leaders are a different ordering, which is itself informative.

The practical rule is a short checklist. Identify the variant. Note whether the figure is vendor-claimed or independent. Ignore gaps under about five points unless the scaffolds are documented and identical. Then ask what each model costs to run, because that difference is real in a way that a few benchmark points may not be.

And if two models land close, pick on price, licence and latency rather than on the third decimal place. Why benchmarks disagree explains why that third decimal was never stable.