Agent Regression Suites: Catching Breaks Before Users Do

You add one sentence to a system prompt to fix a reported bug. The fix works. Three days later someone reports that a different workflow, one nobody touched, has started skipping a confirmation step.

This is the defining failure mode of agent development. Changes are non-local. A prompt is not a function with a signature, so nothing tells you what else depended on the wording you just changed. A regression suite is the only mechanism that gives you that signal, and most teams build one only after the second embarrassing incident.

What a regression suite is not

It is not a benchmark. Benchmarks answer the question of which model is generally stronger. A regression suite answers a much narrower question: does this specific system, with these tools and this prompt, still do the things it did last week?

It is also not a general eval set. The distinction matters for how you grow it. An eval set is curated to cover a space of behaviour. A regression suite is grown by accretion — every time something breaks in production, that scenario becomes a permanent case. The suite is a record of every mistake you have already made and promised not to repeat.

That framing keeps it honest. If you cannot point to the incident that produced a case, the case probably belongs in your broader eval set instead. Evaluating agent reliability covers the wider measurement picture; this article is about the narrow gate.

Freeze everything you can

A regression test is only meaningful if the thing you changed is the only thing that changed. Agents have an unusual number of moving parts, and each one that moves for free makes your signal noisier.

Pin the model version explicitly rather than accepting a floating alias. A silent provider-side update to the model behind a generic name will look exactly like a regression in your own code, and you will spend a day bisecting your prompts before checking the provider changelog. Pinning model versions covers why aliases drift.

Freeze the tool environment too. Record fixtures for network calls, seed the database to a known state, and use a fixed working tree for file operations. An agent that reads a repository will behave differently as that repository evolves, so the repository has to be a fixture rather than a live checkout.

Set temperature to zero and, where the provider supports it, fix the seed. This reduces variance without eliminating it — determinism and seeds explains why identical inputs can still diverge — so plan to run each case several times rather than assuming a single run is authoritative.

Assert on outcomes, not transcripts

The tempting assertion is a string comparison against a known-good response. It is also the assertion that will make your suite useless within a month, because every harmless rewording turns the build red and people start ignoring failures.

Assert on the end state instead. Did the file get written with the correct content. Does the test suite pass. Is the database row in the expected shape. Was the refund tool called with the right amount. These survive rewording and they are the properties you actually care about.

Where the output is genuinely free text, assert on structural properties rather than exact prose: it cites a source, it stays under a length limit, it does not claim a capability the system lacks. Narrow binary checks beat holistic scoring, and they beat exact matching by a wider margin still.

Track the trajectory, not just the verdict

A pass-or-fail column hides the most useful early warning you have. An agent that still succeeds but now takes eleven tool calls where it used to take four is regressing, and it will cross into failure on the next slightly harder input.

Record steps to completion, tokens consumed and wall-clock time for every case, and alert on distribution shifts rather than only on failures. A twenty percent rise in median steps across the suite is a stronger signal than one flaky case going red.

This is also your cost regression check. Agent spend scales with turns, so a planning change that adds two turns to every session is a budget event as much as a quality one. Agent token budgets covers setting the ceilings that make those changes visible.

Handle flakiness explicitly

Some cases will pass four times out of five no matter what you do. Deleting them is wrong, because instability is information. Treating them as hard failures is also wrong, because the build will be red permanently and the gate will be disabled.

Run each case a fixed number of times and record the success ratio rather than a boolean. Then set thresholds per case: a critical path might require five out of five, while a harder exploratory case might require three out of five. The suite fails when a ratio drops below its own threshold, not when any single run fails.

Store the ratios historically. A case that has quietly slid from five-of-five to three-of-five over two months is the kind of slow degradation that no single build would ever catch.

Wire it into CI, or it will not run

A suite people run manually is a suite that runs before releases they were already nervous about. The value comes from running on every change to a prompt, a tool schema or a model configuration.

Keep it fast enough to tolerate. Split into a small blocking tier of ten to twenty critical cases that runs on every pull request, and a full tier that runs nightly. If the blocking tier takes longer than a normal test suite, people will route around it. Running LLMs in CI covers the cost and concurrency mechanics.

Budget for it honestly. A hundred cases run five times each is five hundred agent sessions, and agent sessions are not cheap. That is a real line item, and it is still cheaper than the incident it prevents.

A workable starting point

Take your last five production incidents. Write each one as a case with a programmatic end-state assertion. Pin the model, freeze the fixtures, run each case five times, and record success ratio plus median step count. Wire it to run on any change under your prompts directory, and add a case every time something breaks.

That is a day of work and it converts your incident history into a standing guarantee. Pair it with tracing in production so that new failures arrive as reproducible traces rather than as bug reports written from memory.