Why Benchmarks Disagree, and Which One to Believe

You will regularly find one benchmark placing model A comfortably ahead of model B, and another placing them the other way round. The instinct is to decide one of them is broken, or to quietly adopt whichever supports the decision you already made.

Usually neither is broken. They are measuring different things, and the disagreement is information — it tells you which axis the models differ on, which is more useful than a single ranking would have been.

Task length is the biggest source of divergence

A benchmark where the model reads a problem and produces one answer measures something quite different from one where the model works for forty tool calls without supervision.

The reason is compounding. In a single-turn task, an error affects one answer. In a multi-step task, each step's output becomes the next step's input, so a per-step error rate multiplies rather than averages. Two models three points apart on single-turn accuracy can be twenty points apart on completion rate over twenty steps.

This is why SWE-bench Verified and Terminal-Bench 2.1 produce different orderings. Verified is bounded: read the issue, produce the patch, run the tests. Terminal-Bench measures long unattended sequences. Models optimised for depth-per-step win the first; models optimised for consistency win the second.

So when two benchmarks disagree, check their horizon length first. It explains more disagreements than any other factor, and it maps directly onto whether your own workload is short or long.

Scoring method changes the answer

Mechanically verified benchmarks run a test suite and record pass or fail. Judged benchmarks ask a model or a human whether the output was good.

These reward different things. Mechanical verification rewards correctness and is indifferent to style. Judging rewards clarity, structure and explanation — and is vulnerable to verbosity bias, where longer, more confident answers score better regardless of accuracy.

A model tuned for helpful, well-organised responses does relatively better on judged evaluations. A model tuned for terse correctness does better on mechanical ones. Neither is cheating; they were optimised toward different targets.

Blind human preference, such as Arena voting, is a third category again. It captures what developers actually like to receive, which correlates with usefulness but is not the same as correctness.

Task selection quietly decides a lot

Every benchmark draws from a task pool, and that pool has a character. Predominantly Python, predominantly web, predominantly open-source repositories with good test coverage, predominantly problems whose solutions are short.

A model trained heavily on one distribution will look better on a benchmark drawn from it. That is not contamination; it is the ordinary consequence of measuring a sample rather than a population.

The practical implication is direct: if your work does not resemble the benchmark's pool, the benchmark's ranking does not transfer. A team writing Rust systems code should discount a Python web benchmark heavily, whatever it says.

Contamination is real but overused as an explanation

Benchmark contamination — training data containing the test set — genuinely happens, and it inflates scores on older, widely published benchmarks more than on new or private ones.

It is also the first accusation reached for when a result is inconvenient, usually without evidence. Treat it as one hypothesis among several rather than the default explanation for a surprising number.

The defensible response is not to argue about contamination but to weight private and recently constructed evaluations more heavily, and to keep your own task set, which by construction cannot be in anyone's training data. Benchmark contamination explained covers the mechanism.

Vendor figures and independent figures are different objects

A vendor-reported number is the result of a lab measuring its own model, usually with a harness they built, prompts they tuned, and the freedom to report their best configuration.

None of that is dishonest, and it is not comparable to an independent measurement. Vendor numbers are best read as an upper bound achieved under favourable conditions — useful for knowing what the model can do, not for predicting what it will do in your harness.

When a vendor claim and an independent result disagree, the independent one is usually closer to what you will experience. Vendor-claimed vs independent benchmarks covers reading both.

How to use disagreement productively

Stop asking which benchmark is right and start asking what the disagreement reveals.

If a model leads on bounded benchmarks and trails on long-horizon ones, it has depth but loses consistency across steps — good for one-shot work, risky as an agent. If it trails on mechanical scoring and leads on human preference, it writes pleasant answers that are not always correct — good for explanation, risky for automation.

Then match that profile to your workload rather than to a leaderboard. A model that is third overall but first on the axis you actually use is the right choice, and no aggregate ranking will tell you that.

Finally, keep your own evaluation. Fifteen tasks from your git history, run through each candidate on the same harness, beats every published figure — because it measures the only distribution that matters. How to benchmark LLMs on your own work covers building it.