Regression Testing a Model Upgrade Before It Bites

A model upgrade is a dependency bump where the dependency has no changelog, no semantic versioning and no test suite you can read. It can improve nine things and quietly break the tenth, and the tenth is the one your product depends on.

Teams that treat it like any other release — gate, diff, canary, roll back — have unremarkable upgrades. Teams that swap the model string and deploy find out from users.

Pin first, or nothing else works

You cannot regression test a moving target. If your configuration points at a floating alias, the model can change under you between two identical requests and every comparison you run is meaningless.

Point at the most specific identifier the provider exposes and treat changing it as a code change with a review. Pinning model versions covers the mechanics, and it is the prerequisite for everything below.

Note that pinning does not guarantee byte-identical behaviour. Sampling is stochastic, serving stacks change, and even at temperature zero you should expect variation — determinism and seeds sets out what is actually controllable.

The practical consequence is that a regression test compares distributions, not strings. Any process built on exact output matching will drown in false alarms within a week.

What to capture before you upgrade

You need a baseline recorded on the current model, and it needs to exist before the new version is interesting. Capturing it under time pressure after a release lands is how teams end up shipping blind.

Record three layers. Task outcomes from your evaluation set — pass, fail, and whether a reviewer would have merged it. Behavioural traces: turn counts, tool call sequences, how often the model asked for clarification. And cost: input, output and cached tokens per task.

Also capture a sample of real production traffic with its outputs, scrubbed appropriately. Your evaluation set covers what you thought to test; production covers what users actually do, and the gap between them is where regressions hide.

Store all of it beside the exact model identifier, prompt version and harness commit. A baseline you cannot attribute to a configuration is a number, not a control.

Reading the diff

Run the new version through the same harness and compare task by task rather than in aggregate. An unchanged overall score can hide five newly passing tasks and five newly failing ones, and the five that broke are the story.

Sort by status change and read the newly failing ones first. Then read the newly passing ones, because an unexpected pass sometimes means the model found a shortcut your scoring script accepts and your reviewers would not.

Look hard at the behavioural layer even when outcomes are stable. A version that produces the same results with forty percent more turns has not improved; it has become more expensive, and that shows up in agent cost variance long before it shows up in quality complaints.

Watch for changes in refusal behaviour, verbosity and output formatting. Formatting shifts are the most common cause of a silent integration break, because parsers that were tolerant of one shape are rarely tolerant of two.

Prompts are coupled to versions

This is the part that surprises people. A prompt is tuned, consciously or not, to a specific model's quirks — the phrasing that stops it rambling, the example that fixes its formatting, the instruction that suppresses a failure it no longer has.

So a new version can regress on your prompt while being better on a neutral one. Before concluding the upgrade is worse, test the same tasks with a stripped-down prompt and see whether your accumulated workarounds are the problem.

The same applies to tool schemas. Models differ in how they interpret descriptions and how reliably they fill optional fields, so schema wording tuned to one version may need revisiting on the next.

Budget for prompt work as part of any upgrade rather than treating it as a defect. A version bump that needs two hours of prompt adjustment is normal, not a red flag.

Rolling it out

Move a small share of production traffic to the new version and leave the rest on the pinned baseline. Route by a stable hash of the user or session so the same user sees consistent behaviour, and keep the split adjustable without a deploy.

Monitor a small set of leading indicators rather than waiting for satisfaction to move: retry rate, tool call failure rate, average turns per task, output length, and cost per completed request. These shift within hours, while quality complaints take days.

Keep the rollback path to a configuration change. If reverting requires a deploy, someone will hesitate at the exact moment hesitating is expensive — the same argument behind routing and fallbacks.

Then hold the old pin until the new one has run a full cycle of your real workload, including whatever weekly or monthly job is unusual. The decision rule is simple: no upgrade without a baseline, no rollout without a canary, and no removal of the old pin until a full cycle has passed cleanly. A maintained harness makes all three of those cheap enough to actually do.