Blind Model Comparison: A Method That Survives Bias

Ask a developer to review two patches while telling them which model produced each, and you will learn more about their priors than about the patches. Brand expectation moves judgement hard, and it moves it in both directions — the fashionable model gets credit, the cheap one gets scrutiny.

Blinding costs almost nothing to implement and is the difference between an evaluation you can act on and an expensive way to confirm what you already believed.

Why unblinded review fails

Expectation shapes attention. A reviewer who believes a model is strong reads a marginal patch charitably and moves on; the same reviewer reads the same patch from a model they distrust and finds the flaw. Neither reading is dishonest.

Price anchoring does the same work. A model costing four times as much is assumed to be producing better output, and reviewers unconsciously look for the justification. The effect is strongest exactly where the real difference is smallest.

There is a sunk-cost version too. If your team spent a month integrating something, the evaluation that says a cheaper alternative matches it is unwelcome, and unwelcome results get re-examined until they go away.

This is precisely why Arena voting hides model identities until after the vote. It is the one design decision that makes preference data worth reading.

The protocol

Take fifteen to twenty-five real tasks from your own history, run each candidate on all of them through one identical harness, and write the outputs to files named by a random identifier with the mapping stored separately.

Present them pairwise. For each task, show the reviewer two anonymised outputs side by side, randomising which side each candidate appears on so position bias averages out. Ask a single question: which of these would you rather merge?

Allow a tie, and expect a lot of them. On routine tasks most current models produce equivalent work, and a protocol that forces a winner manufactures differences that do not exist. A high tie rate is a finding — it means the cheaper candidate is sufficient.

Collect a one-line reason with each vote. The votes give you a ranking; the reasons tell you what the ranking is about, and the reasons are usually the more useful output.

Running it without exhausting people

Reviewer fatigue is the practical constraint. Judgement degrades noticeably after about thirty comparisons in a sitting, so cap sessions and spread them across people rather than pushing one reviewer through the whole set.

Use more than one reviewer per pair where you can, and check how often they agree. Two reviewers who agree on eighty percent of pairs are producing a signal; two who agree on fifty-five percent are producing noise, and the fix is a sharper question rather than more votes.

Keep the question binary and concrete. "Which would you merge" produces consistent answers; "which is better quality" does not, because every reviewer weights readability, correctness and convention differently.

If you need finer detail, add two or three separate binary questions — is it correct, does it follow our conventions, would it pass review unchanged — rather than one score out of ten. Rubrics with few concrete criteria are far more reproducible than scales.

Reading the results honestly

Count wins, losses and ties, and compare the gap against the number of tasks. On twenty tasks a three-vote margin is one or two reviewers changing their mind, not a result — do not report a difference smaller than the noise your sample can produce.

Break the results down by task type before you conclude anything. A candidate that loses overall but wins consistently on your database work is telling you to route by task rather than to standardise, and the aggregate would have hidden that entirely.

Put cost next to the ranking. If the leading candidate wins twelve to eight with fifteen ties and costs three times as much per task, the honest conclusion is that the cheaper model is adequate for most of your workload — the reasoning behind when a cheap model is enough.

Then unblind and look for surprises. If a model you assumed was weak performed well, that is the most valuable thing the exercise produced, and it will not survive contact with a team that skipped the blinding.

When blinding is not enough

Blinding controls for identity bias, not for harness bias. If one candidate got a better prompt or a larger context assembly, blind review will faithfully confirm that advantage as though it were a model property. One harness for everyone is the prerequisite.

It also does not control for style preference. Reviewers systematically prefer longer, better-commented output, whether or not it is more correct, which is the same verbosity bias that afflicts judged benchmarks generally.

Where a mechanical check exists, run it first and blind only the residual judgement. A test suite is not biased, and human review is expensive; use it on the questions tests cannot answer.

The decision rule: mechanical scoring where possible, blind pairwise review for the rest, ties allowed, cost in the same table, and no conclusion from a margin smaller than two tasks. Evaluating models on your own codebase covers assembling the task set this runs on.