The Hidden Cost of Failed Runs: Tokens Spent on Nothing

Cost dashboards report total spend. Reliability dashboards report success rate. Almost nobody joins them, and the number that falls out of the join is the one that should drive decisions.

A failed run costs exactly what a successful one costs. The tokens were consumed, the provider was paid, and nothing usable came back.

The figure that matters

Cost per successful run, not cost per run. The relationship is simple: divide total spend by the number of runs that produced usable output, ignoring the rest.

Worked through, a workload with a 60 percent success rate has a true cost per useful result of 1.67 times its nominal per-run cost. At 40 percent it is 2.5 times. A configuration that looks 30 percent cheaper per run and fails twice as often is substantially more expensive per unit of finished work.

This is the arithmetic that decides model tier selection, and it is the arithmetic most model comparisons skip entirely. Price per million tokens tells you almost nothing until it is combined with a success rate measured on your own tasks.

Failures are more expensive than average

The intuition that a failed run costs the same as a successful one is optimistic. Failed runs are typically more expensive.

A run that succeeds finishes when the work is done, often well before any limit. A run that fails frequently runs to the turn ceiling or the budget cap, because the failure mode is the agent continuing to try rather than stopping cleanly.

So the failures cluster at the expensive end of the distribution. Measuring the mean cost across all runs and assuming failures sit at that mean understates them, sometimes by a factor of two.

Agent token budgets covers capping the tail so a single doomed run cannot consume an unbounded amount.

Partial failures are the awkward case

Binary success and failure is a simplification. Real runs land on a spectrum: complete and correct, complete but needing edits, incomplete but with useful progress, and total waste.

The middle categories matter because they are not free either. A run that produces a diff requiring twenty minutes of human correction has consumed tokens and engineer time, and whether that counts as success depends on what the alternative was.

The practical approach is to score outcomes on a small ordinal scale rather than a boolean, and to weight the cost accordingly. A run needing significant rework should count as a partial success at best, or the metric flatters a configuration that produces plausible-looking output nobody can use.

Where the failures come from

Instrument the reason, not just the fact. Four categories cover most of it.

Budget or turn exhaustion — the agent ran out of room. Often a symptom of a task it cannot do with its tools rather than a limit set too low.

Loop detection — the agent got stuck repeating an action. Usually traceable to an uninformative tool error. Detecting agent loops covers catching these early, which converts an expensive full-budget failure into a cheap early one.

Verification failure — the agent finished and the output did not pass a check. This is the healthiest failure mode, because it was caught.

Silent wrongness — the output passed checks and was wrong anyway. The most expensive category by a wide margin, because the cost lands downstream rather than on the invoice.

Failing fast is a cost optimisation

Most spend on a doomed run happens after the point where it became doomed. Detecting failure earlier converts an expensive failure into a cheap one, without changing the success rate at all.

Three checks do most of the work. A loop detector that intervenes on the third identical call rather than at the turn ceiling. A no-progress check that halts when several turns change nothing observable. And an early feasibility check — if the required tool or file does not exist, fail immediately rather than letting the agent explore for twenty turns.

Each of these reduces the average cost of a failure without touching the successes, which is the cheapest kind of optimisation available.

What to build

Log per run: total tokens, outcome on an ordinal scale, failure reason, and turn count. Then report cost per successful run, cost per failed run, and the share of total spend consumed by failures.

That last figure is usually the surprising one, and it is the one that justifies work on reliability to people who think of reliability as a quality concern rather than a budget one. Evaluating agent reliability covers the measurement side.