Vendor-Claimed vs Independent Benchmarks: Reading Both

Every model launch ships with a table of scores, and every one of those tables was produced by the lab that built the model. A few weeks later, independent evaluations land and the numbers are usually a little lower.

The reflex is to call this dishonesty. It almost never is. The two figures are different objects, produced under different constraints, and the useful skill is knowing which one predicts your experience.

Where the gap comes from

The harness is the biggest contributor. A benchmark defines tasks and scoring, not the scaffolding around them, so the lab writes its own runner — retry policy, turn limits, how tool results are formatted, how the file tree is presented. Every one of those choices moves the score, and the lab tuned all of them.

The prompt is second. A vendor has spent months learning what their model responds to and will use a system prompt shaped to it. An independent evaluator uses one prompt across every model so the comparison is fair, which costs each model a few points of its best configuration.

Then there are settings that are easy to omit from a footnote. Reasoning effort levels, sampling temperature, and how many attempts count as a success. GLM-5.2 exposes high and xhigh effort, for instance, and a score is meaningless without knowing which one produced it.

None of this requires bad faith. It is what happens when the party with the most knowledge of a system reports its best achievable result, and it is why the two categories should never be placed in the same column.

What a vendor number is good for

Read it as a capability ceiling: this is what the model can do when someone who knows it intimately sets it up properly. That is real information, especially if you are willing to invest in your own scaffolding.

It is also the only source for architectural facts. Parameter counts, active parameters, context length, licence terms and pricing come from the vendor and there is no independent version of them. Kimi K3 being 2.8T total and 104B active with a 1M context window is a vendor statement, and a perfectly reliable one.

Vendor tables are useful for direction of travel within a family, too. When the same lab reports the same benchmark on a new release under the same harness, the delta is meaningful even if the absolute number is generous.

Where they stop being useful is cross-lab comparison. Two vendor figures on the same benchmark were produced by two different harnesses, and the gap between them contains an unknown amount of scaffolding.

What an independent number is good for

Predicting what you will get from a general-purpose setup. Independent evaluators standardise the harness across models specifically so the comparison survives, which is the property vendor tables lack.

Aggregate indices are the clearest example. Something like the AA Intelligence Index — Kimi K3 around 57, GLM-5.2 around 51, DeepSeek V4 Pro and MiniMax M3 around 44 — is not precise about any single capability, but it is consistent about relative ordering because every model went through the same pipeline.

Independent runs also catch things launch posts do not report: latency under load, refusal behaviour, how the model degrades at long context, and cost per completed task rather than cost per token.

The trade-off is that a standardised harness under-serves models that need specific handling. An independent figure is closer to a floor than a ceiling, which is exactly why reading both is more informative than picking one.

Normalising before you compare

Before two numbers go in the same sentence, check four things: the benchmark variant, the harness, the attempt policy, and the effort or temperature setting. If any of the four is unstated, the comparison is not available and no amount of reasoning fixes that.

Variant confusion causes more bad conclusions than everything else combined. Qwen 3.6 27B at 77.2 percent on SWE-bench Verified and GLM-5.2 at 62.1 percent on SWE-bench Pro are not a ranking, because Verified and Pro are different exams.

Watch for vendor-claimed figures presented without that label. MiniMax M3's 59.0 percent on SWE-bench Pro is a vendor-reported figure, and it belongs in the vendor column even when it appears in a comparison table alongside third-party results.

When the two categories disagree about the same model, the independent result is generally closer to what your team will see, unless you plan to invest in a bespoke harness of your own.

A working procedure

Start with independent aggregates to build a shortlist of three or four candidates, because they are the only figures that survive cross-lab comparison. Use vendor tables to understand what each candidate is capable of when set up well, and to check the hard facts about context, licence and price.

Discount any figure whose harness is not described, and discount older public benchmarks further because of contamination decay. Give more weight to recent, private or rotating evaluations.

Then stop reading tables. Run the shortlist on tasks from your own codebase through one harness you control, and let that decide. Published numbers exist to narrow the field to something you can actually test in an afternoon.