A/B Testing Two Models Without Fooling Yourself

Everyone knows the right way to choose between two models is to test them on your own traffic. Rather fewer people run that test in a way that supports the conclusion they draw from it.

The failure is almost never the plumbing. Splitting traffic is easy. The failure is that language model outputs are high-variance, the effects being measured are small, and the sample sizes people use are an order of magnitude too low.

Decide what you are measuring first

Write down the metric before you route a single request. This sounds like process theatre and it is the step that prevents the most common outcome, which is picking whichever model looks better on whichever chart happened to move.

The best metrics are outcomes you already collect. Did the generated test pass. Did the patch apply cleanly. Did the SQL execute without error. Did the user accept the suggestion, or edit it, or discard it. These are unambiguous, free, and directly connected to the value you care about.

Structural metrics are the second tier: parse success rate, tool-call validity, refusal rate, output length. They are cheap and they catch regressions, but a model can win on all of them and still give worse answers.

Human preference is the third tier. It is the most faithful measure and the most expensive, and it needs blinding — reviewers who know which model produced which output will confirm whatever they already believed.

Paired comparison beats split traffic

The default design sends half of traffic to A and half to B, then compares aggregate metrics. It works, and it wastes an enormous amount of statistical power.

Requests are not interchangeable. Some are trivial, some are impossible, and the variance between requests dwarfs the variance between models. A random split has to overcome that noise with volume alone.

A paired design sends the same request to both models and compares the two outputs directly. Now every comparison controls for request difficulty, and the variance you are fighting is only the model difference. In practice this cuts the sample size you need by a large factor.

The cost is that you pay for both calls. For an evaluation running over a few days on a fraction of traffic, that is almost always the right trade — and if it is not, you are probably running the test too long. Benchmarking on your own work covers assembling the request set.

How much data you actually need

The uncomfortable arithmetic: the smaller the effect, the more samples are required, and the relationship is quadratic. Halving the effect you want to detect quadruples the sample size.

Detecting a large difference — one model succeeding on eighty percent of tasks against sixty — takes a few hundred paired comparisons. Detecting a two-point difference in success rate takes thousands. Most teams run fifty examples and declare a winner.

The practical response is not to compute power curves. It is to ask, before starting, how large a difference would change your decision. If a two-point gain would not justify the migration, do not build a test capable of resolving two points. Test for the effect size that matters and accept that smaller differences are a tie.

And when the result is a tie, treat it as a real result. Two models that perform indistinguishably on your work is extremely useful information: it means you should choose on price, latency or licence instead. Cost-adjusted scoring is the tie-breaker.

Confounds that quietly ruin the comparison

The prompt is the biggest one. A prompt tuned over months against model A is not a neutral instrument for testing model B. B will underperform, and the result measures your tuning rather than the models.

There is no perfect fix, only honest options. Test with a deliberately plain prompt that favours neither, or spend comparable tuning effort on both, and say which you did when reporting the result.

Sampling settings are the next confound. Compare at the same temperature, and preferably at low temperature, or you are measuring randomness. Temperature and top-p covers what each knob does to variance.

Time is the third. Traffic composition changes across a week, so a test that runs A on Monday and B on Thursday has confounded model with day. Run them concurrently, always.

Reading the result honestly

Look at the distribution, not the mean. A model with a slightly lower average success rate but far fewer catastrophic failures is often the better production choice, and the mean hides that entirely.

Segment before concluding. Aggregate parity frequently conceals that one model is clearly better on long-context requests and clearly worse on short ones. That finding is more valuable than the overall winner, because it points at routing rather than replacement. Multi-model strategies covers acting on it.

Check cost and latency alongside quality in the same run. You are already collecting both, and a model that wins on quality while doubling time to first token may still lose the decision. Latency-adjusted scoring makes that explicit.

A workable protocol

Pick one outcome metric you already collect. Decide the effect size that would change your decision. Run both models on the same requests, concurrently, at the same temperature, with a prompt that is not tuned to either. Collect until you have enough paired comparisons for that effect size, then stop.

Report the distribution and the segments, not just the mean. If the difference is inside your threshold, call it a tie and decide on price, latency or licence.

Then keep the harness. The next model ships in a few months, and a comparison you can re-run in an afternoon is worth far more than a one-off result. When to switch models covers what to do with it.