Evaluating Prompt Changes Without Fooling Yourself

The standard prompt evaluation is running the new version on three inputs, liking the outputs, and shipping. It fails for a specific reason: the outputs you looked at were chosen by the same intuition that wrote the change, and three samples cannot distinguish a real improvement from run-to-run noise.

You do not need a research setup to do better. You need a fixed evaluation set, paired comparisons, and enough discipline to accept a null result.

Build the set from failures, not from happy paths

An evaluation set of typical inputs will tell you the model is fine, because it is fine on typical inputs. The information lives at the edges.

Harvest the set from production. Every time someone reports a bad output, add that input to the set with the correct answer attached. Every time you fix a bug by editing a prompt, the triggering input becomes a permanent regression case. Within a couple of months you have fifty to two hundred cases that all represent something that actually went wrong, which is worth more than a thousand synthetic ones.

Keep a slice of ordinary inputs too, sized at maybe a quarter of the set. Their job is to catch the change that fixes the edge case by wrecking the common case, which is the most frequent way a prompt edit does damage.

Compare paired, on the same inputs, at the same time

Run both prompt versions on the identical input set in the same session, against the same pinned model ID. Comparing against numbers recorded last month means you are also measuring whatever the provider changed in between, which you cannot separate.

Pairing matters statistically as well as practically. If version A and version B are evaluated on the same cases, you can look at per-case wins and losses rather than at two aggregate scores, and per-case differences have far less variance than the aggregates do. A change that wins on eight cases and loses on two is a much clearer signal than a score moving from 0.71 to 0.74.

Report the disagreements, not just the totals. The two cases where the new version lost are the ones that tell you what the change actually did, and they usually reveal that the edit traded one behaviour for another rather than improving anything.

Sampling noise is bigger than you think

At any temperature above zero, the same prompt on the same input gives different outputs. If your metric moves by three points and your run-to-run variance is four points, you have learned nothing.

Measure the noise floor before you measure the change. Run the unmodified prompt through the set several times and record the spread of the metric. That spread is the minimum effect size you can detect. Anything smaller needs more samples or a lower-variance metric.

Setting temperature to zero reduces but does not eliminate this. Batching, hardware scheduling and floating-point non-associativity mean identical requests can still diverge, which is covered in more detail in why temperature zero does not give you reproducible output. Treat determinism as a variance reduction technique, not as a guarantee.

Choose a metric that can be wrong

The best metric is one that fails when the output is bad, and that you did not design after seeing the outputs.

For structured tasks this is easy and you should exploit it. Extraction gives you per-field exact match. Classification gives you accuracy and a confusion matrix. Code generation gives you a test suite. Anything where a program can check the answer should be checked by a program, because programs do not get tired or generous.

For open-ended tasks, decompose before you resort to judgement. A summary can be checked for whether every named entity in the summary appears in the source, whether required sections are present, and whether length is in range. Those are three cheap deterministic checks that catch a large fraction of real problems, and they leave a much smaller residue for a human or a model judge.

Model-as-judge, used carefully

Using a model to grade outputs scales, and it introduces its own biases that you have to control for.

Judges prefer longer answers. They prefer answers in their own style. In pairwise comparison they show position bias, favouring whichever candidate appears first. The first two you handle by giving the judge a rubric with explicit criteria rather than asking which is better. The third you handle by running each comparison twice with the order swapped and discarding pairs where the judge contradicts itself, which also gives you a free self-consistency rate.

Calibrate the judge before you trust it. Take fifty cases, grade them by hand, and check the judge agrees. If agreement is poor, fix the rubric rather than the judge. And never use the same model, with the same prompt, as both generator and judge on a task where the failure is a reasoning error, because the judge shares the error.

score_ab = judge(a, b)
score_ba = judge(b, a)
if score_ab != flip(score_ba):
    unstable += 1        # report this rate alongside the result

One change at a time, and write down the prediction

Bundled prompt edits are unattributable. You added an example, tightened the instruction and reordered two sections; the score went up two points; you have no idea which of the three did it, and one of them may be hurting.

Change one thing, run the set, record the result with the version identifier. This is slow, and it is the only way to build up knowledge that transfers to the next prompt rather than folklore that does not.

Write your prediction before the run. "This should fix the null-handling cases and not affect anything else" is a falsifiable claim, and being wrong about it teaches you something about the model. Vague expectations are unfalsifiable and teach you nothing. Attach the result to the prompt version so the history is legible later, using the identifiers described in versioning prompts so outputs are traceable.

Costs and latency are part of the result

A prompt change that improves accuracy by two points and adds fifteen hundred tokens to every request is not obviously a win. Record tokens in, tokens out and wall-clock latency alongside the quality metric, every time.

Long chain-of-thought instructions are the usual culprit. They often help, and they multiply output tokens, which is the expensive direction on most pricing. Whether the trade is worth it depends on your volume and on the cost of a mistake, and the arithmetic is laid out in how input and output tokens are priced differently.

A workable loop

Fix the evaluation set and pin the model. Measure the noise floor. Make one change. Run both versions paired on the same set. Look at the per-case wins and losses, not the aggregate. Check cost and latency moved acceptably. Record the result against the version identifier, then ship or revert.

The most valuable habit in that list is accepting a null result. Most prompt edits do nothing measurable, and knowing that stops you accumulating a prompt full of superstitions that each cost tokens forever. The example set deserves the same scrutiny, which is why pruning few-shot examples that no longer earn their tokens is usually the first thing worth testing.