What a Model Upgrade Costs Beyond the Per-Token Rate

A new model ships, the rate card is lower than what you are paying, and the upgrade looks like free money. Change the model string, watch the bill fall.

It frequently does not fall, and occasionally it rises. The per-token rate is one term in an expression with several others in it, and an upgrade moves most of them at once — output length, tool-call frequency, retry rate, cache hit rate and latency. This is an inventory of what actually changes and how to price it before you roll.

Output length is the term that moves most

Output tokens are typically priced several times higher than input tokens, so a model that is more thorough costs more per request even at a lower rate.

Newer models are often tuned to be more complete: more explanation, more structure, more caveats. On a workload where you wanted three lines and got twelve, a thirty percent rate cut is wiped out and then some. The effect is largest on short-output tasks, where a fixed amount of added preamble is a large proportional increase.

Reasoning behaviour is the sharper version of this. If the new model produces internal reasoning tokens, or produces more of them, those are billed output tokens and they do not appear in the response you read. A model that reasons more can be genuinely better and substantially more expensive per call. Token cost of reasoning models covers the accounting.

Measure this directly: run your evaluation set through both models and compare mean and p95 output tokens per task, not just quality scores. It is a five-minute measurement that changes the conclusion more often than any other.

Prompt drift: your instructions were tuned for the old model

Production prompts accumulate corrections. A line that stops the old model padding, an example that fixed a format it kept getting wrong, an emphatic instruction it needed repeating.

Against a new model those lines are at best inert and at worst harmful. Instructions written to suppress a tendency the new model does not have can push it into odd behaviour, and the classic symptom is a prompt that scores worse after the upgrade until someone deletes half of it.

The work is not writing a new prompt; it is finding which of the old corrections are now unnecessary, which is iterative and needs evaluation between each change. Budget it per prompt. Prompt regression testing is what makes this tractable rather than guesswork.

Tool-calling behaviour changes quietly

If your workload calls tools, the upgrade will change how often and in what pattern.

Common shifts: emitting several tool calls in parallel where the old model emitted them serially, calling a tool more eagerly rather than answering directly, or being stricter about required arguments and therefore asking clarifying questions your loop was not written to handle. Each changes both cost and success rate.

Parallel calls in particular can cut wall-clock latency and raise token spend at the same time, because more tool results come back into context. If your agent loop assumed one call per turn, it may also silently drop the extras. Parallel tool calls covers what to check.

The metric to watch is tool calls per completed task, before and after. If it moves more than slightly, your cost model has changed regardless of the rate card.

Retries and first-attempt success

A better model usually raises first-attempt success, and that is a real saving that rate cards do not show. Fewer retries means fewer duplicated requests on exactly the hard items that cost the most.

But it can go the other way. If your validation is tuned to the old model's output shape — a particular JSON layout, a heading it always produced, a phrasing your parser expects — the new model can fail validation on outputs that are perfectly good, and your retry rate spikes on a model that is objectively better.

Check the failure reasons, not just the failure rate, in the first week after any upgrade. Parser-shaped failures are cheap to fix and expensive to leave in place. The hidden cost of retries covers the arithmetic of what a shifted retry rate does to a bill.

Cache hit rate and latency both shift

Prompt caching discounts depend on stable prefixes. Change the model and any prefix you touch during re-tuning invalidates existing cache behaviour, and the caching rules themselves may differ between model versions.

The visible symptom is a cost increase in the first days after a rollout that then partially recovers as caches warm. Teams frequently misread this as the upgrade being more expensive than it is, then misread the recovery as an optimisation working.

Latency moves too, and not always downward. A larger or more reasoning-heavy model raises time to first token and total completion time, which matters if anything downstream has a timeout or if a human is waiting. Re-check your timeout settings against the new p95 rather than assuming the old ones hold. Timeout tuning for LLM calls covers the adjustment.

Regression testing is the actual price of admission

Everything above is only detectable if you run the same tasks through both models and compare. Without that, an upgrade is a change with unknown effects on quality and cost simultaneously.

The minimum viable suite is a fixed task set drawn from real production inputs, with either expected outputs or graded criteria, plus per-task recording of input tokens, output tokens, tool calls and wall-clock time. Quality scores alone will not tell you the bill went up.

Running it is cheap. Building it the first time is not, and that build cost is properly attributed to the first upgrade rather than treated as overhead. Regression testing model upgrades covers the workflow; after the first time, each subsequent upgrade is a day rather than a fortnight.

Why a cheaper model can raise total spend

Put the terms together. Total cost per completed task is roughly: input tokens times the input rate, plus output tokens times the output rate, multiplied by expected attempts, plus whatever tool-call round trips add to context on each attempt.

A model with a rate twenty percent lower that produces sixty percent more output tokens on your workload is more expensive, because output dominates. One that makes an extra tool call per task carries the cost of that result back into context on every subsequent turn. One whose outputs fail your parser more often multiplies everything by a higher attempt count.

Conversely a more expensive model can be cheaper overall if it raises first-attempt success materially or shortens a multi-step agent run. This is the same calculation as deciding when a cheap model is enough, run in the other direction.

The rate card is a per-unit price for a unit whose quantity changes when you change models. That is the whole reason it misleads.

A rollout that does not surprise you

Run the regression suite first and record tokens and tool calls alongside quality. If output length moved materially, re-tune the prompt before you compare costs, since half the increase is usually removable.

Then roll to a small traffic percentage and watch cost per completed task rather than cost per request — the second can fall while the first rises. Give caches a few days to warm before drawing conclusions, and check failure reasons rather than failure counts.

Keep the old model reachable behind a routing rule until you have a full billing cycle on the new one. Upgrades are usually right, and the ones that are not are much cheaper to reverse when reversing is a configuration change. Pinning model versions is what makes that possible in the first place.