Cost & PricingThe Hidden Cost of Retries: Paying Twice for the Same Answer
Retry logic makes pipelines reliable and quietly doubles spend on the items that need it. How to measure what retries actually cost you.
Every production LLM pipeline retries. Validation fails, the output does not parse, a tool call is malformed, the provider returns a transient error. Retrying is correct and it is not free, and the cost is almost never measured because it hides inside an aggregate.
Why it is invisible
Cost dashboards show total spend. Success dashboards show final success rate. Neither shows how many attempts produced that success.
A pipeline retrying twenty percent of items looks identical from the outside to one retrying two percent, provided both end up succeeding. The difference only appears if you record first-attempt success separately, which most instrumentation does not.
So the first move is not optimisation, it is measurement. Log attempt counts per item and compute the distribution, not the average.
The arithmetic
With a first-attempt success rate of p, expected attempts per item is roughly 1/p if failures are independent. At 95 percent that is about 1.05 attempts — negligible. At 80 percent it is 1.25. At 50 percent it is two, meaning you are paying double.
Failures are usually not independent, which makes it worse. An item that failed once is more likely to fail again, because whatever made it hard has not changed. Retrying the identical prompt against the identical input frequently reproduces the identical failure.
The practical consequence is that the tail is heavier than the independence assumption predicts. A small fraction of items can consume a disproportionate share of total spend, and averages conceal this completely.
Retries that cannot succeed
The most wasteful category is retrying something structurally impossible.
If a prompt asks for a field that is not present in the input, no number of attempts produces it. If a schema is ambiguous, retrying re-rolls the same ambiguity. If a document is genuinely unparseable, the model will keep failing.
These consume the full retry budget and end in failure anyway, at maximum cost. They are worth identifying specifically: look for items that exhaust retries rather than items that eventually succeed, and investigate what they have in common. It is usually a systematic problem with a systematic fix.
Change something between attempts
An identical retry is close to a coin flip on the same coin. A retry that includes information about the failure is a correction.
Feeding the validation error back — naming the field and what was wrong with it — converts the second attempt into a different, easier task. Success rates on the second attempt improve substantially, which cuts the number of third attempts.
Other useful variations: lower the temperature, since a failed structured output often means sampling wandered; simplify by splitting one complex request into two simple ones; or escalate to a stronger model rather than repeating with the same one. Agent error recovery patterns covers the general shape.
Retry cost interacts with model choice
This is where the arithmetic changes conclusions.
A model a third of the price with an 80 percent first-attempt rate against one at full price with a 97 percent rate: the cheap model costs roughly 1.25 attempts at a third of the price, which is still well under half. It wins.
But that only holds if failures are detectable. If a wrong answer passes validation and reaches production, you have not saved money — you have bought a defect at a discount. The cheap-model-plus-retries strategy is entirely dependent on having a check that catches failure.
So before switching to a cheaper tier, ask whether your pipeline can tell when it received a bad answer. If it cannot, retries are not a safety net and the cheaper model is not cheaper. When a cheap model is enough covers making that call.
What to instrument
Four numbers, all cheap to collect and none available by default.
First-attempt success rate, which tells you whether retries are a rare correction or a routine part of the pipeline. Attempt distribution rather than mean, so the tail is visible. Exhausted-retry rate, which isolates the structurally impossible cases. And tokens spent on failed attempts as a share of total, which is the actual answer to what retries cost.
That last figure is often surprising the first time anyone computes it, and it is the one that justifies spending an afternoon on prompt or schema fixes that raise first-attempt success.
Common questions
How much do retries typically cost?
At a 95 percent first-attempt rate, about 5 percent overhead. At 80 percent, around 25 percent. At 50 percent you are paying double — and because failures are correlated rather than independent, the real tail is heavier than that.
Should I retry with the identical prompt?
No. An identical retry usually reproduces the identical failure. Include the validation error, naming the field and what was wrong, which turns the second attempt into a correction rather than another coin flip.
Does a cheap model plus retries beat an expensive one?
Only if failures are detectable. If a wrong answer passes validation and reaches production, you have bought a defect at a discount rather than saving money. The strategy depends entirely on having a real check.
