Test-Time Compute: Buying Accuracy With Tokens Instead of Size

For most of the last decade, better output meant a bigger model. Train more parameters on more data, and accuracy improves. That axis still works, but it is expensive and slow to iterate on.

Test-time compute is the other axis. Keep the model fixed and spend more computation when the request arrives — generate more tokens, sample multiple times, verify and revise. On a range of tasks this buys accuracy that would otherwise require a substantially larger model.

Why extra computation helps at all

A model does a fixed amount of work to produce each token. It cannot think harder about a difficult token than an easy one; the forward pass is the same size either way.

So a problem needing more computation than one forward pass provides cannot be solved in one token. Demanding an immediate answer forces a guess. Generating intermediate tokens gives the model more forward passes, each taking the previous output as input, spreading the computation across many passes instead of compressing it into one.

That is the entire mechanism, and it explains the shape of the benefit: it helps most where the answer must be built up in sequence and least where the answer is a lookup. Chain of thought explained covers the prompted version of this.

The three ways to spend it

Sequential. Generate a longer reasoning trace before answering. One pass, more tokens. This is what a reasoning model does by default, and it is the cheapest form to implement because it requires no orchestration.

Parallel. Sample several independent answers and aggregate them, typically by majority vote. Latency stays close to a single call if you run them concurrently, but you pay for every sample. Self-consistency decoding covers the aggregation rules.

Iterative. Produce an answer, evaluate it against a check, revise. This is the most powerful form when a real verifier exists — tests that run, a compiler, a schema validator — and the weakest when the only verifier is the model grading its own work.

The verifier is what determines the ceiling

Test-time compute converts computation into accuracy only if something can distinguish a better attempt from a worse one. With a strong verifier, generating twenty candidates and keeping the one that passes is close to free accuracy.

Code is the favourable case because tests are a genuine external verifier. An agent that writes a patch, runs the suite, reads the failure and revises is doing exactly this, and the gains are real. It is a large part of why coding agents work at all.

Self-critique without an external signal is much weaker. A model asked to check its own answer frequently agrees with itself, including when it is wrong, because the same weights produced both the answer and the assessment. Treat self-review as a mild filter, not a verifier. Evaluating agent reliability covers measuring the difference.

Where the returns stop

Returns diminish, and on many tasks they diminish quickly. Doubling reasoning length on a problem the model was going to get right anyway buys nothing and costs double.

Worse, extended reasoning on an easy question sometimes converges on an elaborate wrong answer where a direct response would have been correct. Extra computation gives the model room to talk itself out of a right instinct.

There is also a hard limit: no amount of inference-time computation supplies knowledge the model does not have. A question about your internal API cannot be reasoned into a correct answer. That needs retrieval, not effort. RAG versus long context covers supplying the missing facts instead.

What it costs you

Reasoning tokens are output tokens and are billed as such, which is the expensive side of the price sheet. Kimi K3 lists $3 in and $15 out per million; GLM-5.2 lists roughly $1.40 in and $4.40 out. A reasoning-heavy request can emit many times more tokens than appear in the visible response.

Latency is the harder cost. Sequential reasoning is serial by nature, so a long trace directly delays the first useful token. Parallel sampling is friendlier to wall-clock time but multiplies spend and can hit rate limits. LLM inference latency explained covers where the time goes.

Budget for this explicitly rather than discovering it. An agent that reasons at every step is the classic way to be surprised by a bill. Agent token budgets covers capping it per task.

The effort dial is the practical control

Several models now expose a reasoning effort setting. GLM-5.2 offers two levels, high and xhigh. That dial is usually the right interface: raise it for genuinely hard steps and lower it for routine ones.

This captures much of the benefit of routing between a small and a large model without maintaining a second integration. The routing logic is a per-request parameter rather than a separate provider and key. Model routing and fallbacks covers when a full router is worth it instead.

If your provider exposes no dial, you can approximate it with prompt structure: a request for a brief direct answer versus an explicit instruction to enumerate constraints and evaluate options. The instruction shapes how much intermediate computation the model performs.

A decision rule

Spend test-time compute where the task is multi-step and a verifier exists. Skip it where the task is retrieval, classification or formatting, and cap it where cost per request matters more than the last few points of accuracy.

Then measure. Run your task set at two effort levels and compare accuracy against token spend. If accuracy is flat, you are paying for computation that buys nothing — a common outcome, and one almost nobody checks. Reducing token usage covers finding that waste.