max_tokens and Truncation: Why Your Output Stops Mid-Sentence

A response that stops in the middle of a word looks like a bug in the model. It almost never is. It is a limit somewhere in the request path being reached, and the generation loop halting because it was told to.

The frustrating part is that a truncated response usually returns a 200 status code. Nothing errors. You get a syntactically broken JSON object or half a function, and unless you check the stop reason you have no idea why.

What max_tokens actually controls

The parameter caps the number of tokens generated in the response. It does not cap the total request, it does not reserve space, and it does not make the model plan a shorter answer.

That last point causes most of the confusion. Setting a cap of 500 does not instruct the model to write a 500-token answer. The model writes whatever it was going to write, and generation is cut off when the counter hits 500. If the natural response was 900 tokens, you get the first 500 and a hard stop wherever that lands.

If you want a short answer, ask for one in the prompt. Use the cap as a safety rail against runaway generation, not as a length control. Those are different jobs and conflating them produces exactly the mid-sentence cutoff people complain about.

Input and output share the window, but not the limit

The context window is the total budget for input plus output. On a model with a 256K window, a 250K-token prompt leaves roughly 6K for the response no matter what you set the cap to.

Providers handle this inconsistently. Some reject the request. Some silently generate until the window is exhausted. Some accept a cap larger than the remaining space and truncate at whichever limit binds first. Context windows explained covers the accounting.

Separately, many models publish a maximum output length well below their context window. GLM-5.2 ships a 1M context and a 128K maximum output. Asking for more output than the model can produce in one response is a request-shaping error, not something to solve with retries.

Read the finish reason, always

OpenAI-compatible APIs return a field on each choice indicating why generation stopped. A natural completion reports one value; hitting the token cap reports another, usually a variant of "length".

Checking this field is the single highest-value thing you can add to an LLM integration. It converts a class of silent corruption into an explicit, handleable condition. Without it, a truncated JSON payload arrives at your parser as a generic syntax error and you spend an afternoon suspecting the model.

In a streaming context the same information arrives on the final chunk. If you are assembling deltas and never inspecting the terminal event, you are discarding the one signal that tells you the output is incomplete. Streaming and server-sent events covers the event shape.

Structured output is where it hurts most

Prose degrades gracefully under truncation. A cut-off paragraph is still readable and a human notices immediately. Structured output does not degrade gracefully at all.

A JSON object truncated at the token cap is not valid JSON. Every downstream consumer fails, and the failure looks like a parsing problem rather than a length problem. If you are generating structured data, the finish-reason check is not optional. Structured outputs and JSON mode goes into the schema side.

Tool calls have the same property. A truncated arguments object means the call cannot be dispatched, and in an agent loop that turns into a retry, then another retry, each one paying full input cost for a request that was always going to exceed the cap. The hidden cost of retries covers where that money goes.

Reasoning tokens count against the cap

On models that reason before answering, the internal reasoning consumes output tokens. Some providers count them against max_tokens directly.

The visible symptom is bizarre: you set a generous cap, the model thinks extensively about a hard problem, and you receive an empty or nearly empty response. The budget was spent before the answer started. On an easy prompt the same configuration works fine, which makes it look intermittent.

If you use a reasoning model with a cap, size the cap for reasoning plus answer, not answer alone. Where an effort level is exposed, lowering it is usually the better fix than raising the cap. Reasoning models explained covers the trade-off.

Truncation on the input side is quieter

Output truncation at least leaves visible damage. Input truncation often does not. A pipeline that chunks documents to a fixed token size, computed with the wrong tokenizer or an approximation, silently drops the tail of every chunk that overflows.

The model then answers confidently from partial input, and nothing in the response indicates that material was missing. This is a genuinely nasty failure because the output is fluent and plausible. It is also common in retrieval pipelines assembled quickly. Context length versus effective context covers the related gap between advertised and usable capacity.

A short checklist

Set an explicit cap on every request rather than relying on a provider default you have not read. Defaults vary between providers and change between versions.

Check the finish reason on every response and treat a length stop as a first-class error path, not a log line. Decide deliberately whether that path retries with a larger cap, asks the model to continue, or fails loudly.

Control length through the prompt and validate the output shape independently. If you need a bounded response, say so in words and verify what came back — the cap is the backstop, not the mechanism.