Cost Per Test Suite: Writing Tests Versus Fixing Them

Teams budget for AI-generated tests as if it were one activity. It is two, and they differ by an order of magnitude in cost.

Writing a new test file is a single generation against a known input: the model reads a module, emits a file, done. Fixing a suite that has gone red is an agent loop with an unknown number of iterations, each of which resends everything before it. The first is a line item. The second is the thing that shows up on the invoice.

Writing tests is bounded work

The cost of generating a test file is roughly the source module plus the conventions you supply plus the output. All three are knowable before you start.

Say the module under test is 400 lines, you include two existing test files as style examples, and the model emits 200 lines of tests. That is a few thousand input tokens, maybe a thousand output tokens, and one call. Multiply by the number of modules and you have a defensible estimate for the whole exercise.

The estimate holds because there is no loop. Nothing feeds back, nothing accumulates, and a bad result costs one retry rather than an open-ended investigation. Generating tests with LLMs covers making the output worth keeping; the point here is only that its cost is predictable.

Fixing a failing suite is unbounded work

Now the shape changes completely. The model runs the suite, reads the failure output, forms a hypothesis, edits code or test, runs again, and repeats until green or until it gives up.

Each iteration appends the full test runner output to the transcript, and the whole transcript is resent on the next call. Cost grows roughly with the square of iteration count rather than linearly, which is why a debugging session that takes twelve turns instead of four costs far more than three times as much. Cost per agent run works through that quadratic in detail.

The termination condition is also outside your control. A syntax error takes one turn. A race condition in an async test can take twenty, and may not terminate at all — the model patches a timeout, the test passes once, it fails again on the next run, and the loop restarts on what looks like a fresh problem.

The re-read tax on large test files

There is a specific pathology worth naming because it is both expensive and easy to fix.

Test files grow large. A 3,000-line integration test file is common, and a model that opens it to fix one assertion pays for all 3,000 lines. That would be tolerable once. The problem is that agents frequently re-read the same file across turns — after an edit, to check context, or because a compaction step dropped it from the transcript and the next tool call fetched it again.

Four reads of a 3,000-line file is 12,000 lines of input for one assertion change, and every one of those reads also sits in the transcript being resent on each subsequent turn. The multiplier compounds with the quadratic above.

Two mitigations, in order of effect. Prefer tools that read a line range or a single test case rather than a whole file. And split oversized test files along the same boundaries you would split source files — this is worth doing for human reviewers anyway, and it cuts the per-turn read cost proportionally. The hidden cost of context bloat covers the general version of this multiplier.

Failure output is the other quiet expense

A test runner in verbose mode can emit thousands of tokens per run: stack traces through framework internals, full object diffs, log lines from every passing test alongside the failing one.

All of it enters the transcript, and most of it carries no signal. The model needs the failing test name, the assertion, the expected and actual values, and the top few frames of the stack that live in your code.

So run the failing tests only, not the suite, once you know which ones are red. Turn off progress chatter. Truncate diffs of large structures to the differing fields. Teams that do this routinely see per-turn input drop by more than half on debugging sessions, which is a larger saving than any model swap available to them.

Tests the model wrote are the ones it struggles to fix

A generated suite often contains assertions that encode current behaviour rather than intended behaviour — the test was written by reading the implementation, so it asserts whatever the implementation did.

When such a test fails after a legitimate change, there is no correct answer available from the code alone. The model cannot tell whether the change is a regression or an intended update, so it tends to edit the test until it passes, which is both wrong and expensive: it burns iterations on a question that only a human with intent can settle.

This is the strongest argument for reviewing generated tests properly at the point they are created. A tautological test is not merely useless, it is a recurring cost centre every time it goes red. Choosing a model for test generation covers which models are least prone to it.

Putting a ceiling on it

Bound the loop explicitly, because it will not bound itself.

Cap iterations. Five or six turns, then stop and hand back what was learned. A model that has not fixed a test in six turns is usually missing information that no further turns will supply. The hidden cost of retries explains why repeated attempts on the same input correlate rather than resetting the odds.

Fail fast on the first red test. Fixing one test at a time keeps the transcript small and the hypothesis narrow. Handing the model twenty failures at once produces a large context and a scattered investigation.

Reset instead of compacting. When a debugging session goes long, a fresh session with a written summary of what has been ruled out is usually cheaper and more effective than continuing to carry twelve turns of dead ends. Context compaction strategies covers when each applies.

What to measure

Log two separate figures: cost per generated test file, and cost per red-to-green fix. Keep them apart, because averaging them hides the only one you can act on.

Then watch the iteration distribution on fixes. If the median is two turns and the ninety-fifth percentile is twenty, your budget problem is entirely in that tail, and the fix is a cap plus better failure output rather than a cheaper model.