Chain of Thought: Why Thinking Out Loud Actually Helps

Asking a model to work through a problem step by step before answering improves accuracy on certain tasks, sometimes dramatically. It also inflates token usage and latency, and on many tasks it does nothing at all.

Knowing which regime you are in requires understanding why it works, which is less mystical than it is usually made to sound.

The mechanism: computation per token is fixed

A model does a fixed amount of computation to produce each token. It cannot think harder about a difficult token than an easy one — the forward pass is the same size either way.

So if a problem requires more computation than fits in one forward pass, the model cannot solve it in one token. Demanding an immediate answer forces a guess.

Generating intermediate reasoning gives the model more forward passes. Each step written down becomes input to the next step, so the computation is spread across many passes instead of compressed into one. The visible text is a side effect; the extra computation is the point.

This framing predicts the behaviour well. It explains why chain of thought helps most on multi-step problems, why it helps least on retrieval-style questions, and why the intermediate steps do not need to be shown to a user to be useful.

Where it genuinely helps

Tasks with sequential dependency, where step two requires the result of step one.

Arithmetic and unit conversion. Logical deduction with several premises. Code tracing, where you must follow state through control flow. Planning, where later choices depend on earlier commitments. Anything where a human would reach for paper.

The common property is that the answer cannot be computed in parallel from the input. It has to be built up.

Where it wastes tokens

Tasks that are essentially lookup or pattern recognition.

Factual recall, simple classification, format conversion, straightforward extraction. If a knowledgeable person would answer instantly without deliberating, reasoning steps add cost and latency for no accuracy gain — and occasionally hurt, by giving the model room to talk itself out of a correct initial instinct.

That last failure mode is real and underappreciated. Extended reasoning on an easy question sometimes converges on an elaborate wrong answer that a direct response would have got right.

Reasoning models changed the economics

Models trained specifically to reason before answering — with the reasoning process itself optimised during training — do this better than prompted reasoning ever did, and often hide the intermediate tokens.

You still pay for those hidden tokens. A reasoning model on a hard problem can emit many times more tokens than appear in the response, and that shows up on the bill rather than in the output. Budget for it, and measure it rather than assuming.

Several models now expose an effort level. GLM-5.2 offers two. That dial is the practical control: high effort for genuinely hard steps, low for routine ones. It captures much of the benefit of routing between models without a second integration. Reasoning models explained covers the training difference.

Practical guidance

Do not apply it uniformly. Blanket "think step by step" across a whole pipeline inflates cost on every request including the easy majority. Apply it where the task shape justifies it.

Ask for structure, not just length. "List the constraints, then evaluate each option against them, then choose" outperforms a vague instruction to reason, because it directs the intermediate computation rather than merely permitting it.

Separate reasoning from output. Have the model reason in a marked section and give the final answer in a defined format afterward. This keeps parsing reliable and lets you discard the reasoning when storing results.

Measure it. Run your task set with and without. If accuracy is unchanged, you are paying for tokens that buy nothing — which is common and rarely checked. Reducing token usage covers finding this kind of waste.

The counterintuitive part

The reasoning text does not have to be correct for the answer to improve. Models sometimes produce flawed intermediate steps and still reach the right conclusion more often than they would have without them.

That fits the computation framing rather than the explanation framing: the value is in the extra forward passes, not in the reasoning being a valid argument. It also means you should not treat visible reasoning as an audit trail. It is a plausible narrative that accompanied the computation, not a faithful record of it — which matters if you were planning to show it to users as justification.