Stop Sequences Explained: Ending Generation on Purpose

Stop sequences are the least interesting parameter on the completions API and the one most likely to cause a silent data-loss bug. They are simple in concept — halt generation when this text appears — and every part of how they are implemented has an edge case attached.

How generation normally ends

A model generates one token at a time with no inherent notion of finishing. Left alone it would continue until it hit the context limit.

Two things stop it under normal conditions. The model can emit a special end-of-turn token, which it learned during instruction tuning to produce when a response feels complete, and the server halts on seeing it. Failing that, the max-tokens limit cuts generation off wherever it happens to be.

Stop sequences add a third mechanism, controlled by you rather than by the model or the budget. You supply a short string, and the server terminates generation as soon as that string appears in the output.

Where they are applied

The important detail is that stop sequences are matched on decoded text, not on tokens. The server accumulates generated tokens, converts them to text, and checks whether any stop string has appeared.

This matters because your stop string almost certainly does not align with token boundaries. If you stop on </answer>, the model may produce that as one token, or as four, or as part of a token that also contains the preceding space. Matching on text rather than tokens is what makes stop sequences work at all despite that.

It also means the model was never prevented from generating the stop string. It generated it, and the server threw away everything from that point. The difference shows up in billing: you are typically charged for the tokens the model produced, including the ones discarded.

The three behaviours that vary by provider

First, whether the stop text is included in what you receive. Most providers exclude it, so if you stop on </json> your response ends just before the tag. If you were relying on the closing tag to parse, you will be parsing something malformed and will not immediately see why.

Second, how many stop sequences are permitted. Four is a common cap and it is low enough to hit. Design your delimiters so one string covers the case rather than enumerating variants.

Third, and most consequential: whether whitespace and case are handled the way you assume. A stop sequence of Observation: will not fire on observation:, and a model asked to produce a labelled block will vary its capitalisation more than you expect.

The check almost everyone skips

Every response carries a field indicating why generation ended — commonly finish_reason or an equivalent. It distinguishes a natural end, a stop sequence firing, and the token limit being hit.

Truncation by token limit is the dangerous one, because the output looks like ordinary text. A JSON object cut off mid-string throws a parse error you will notice. A summary cut off mid-sentence gets stored, indexed and served to a user, and nobody finds out.

Check the field on every call and treat a length-limit stop as an error, not as a result. This is a five-line change that catches a whole class of bugs, and it is missing from most integrations. Debugging LLM API errors covers the other response fields worth inspecting.

Streaming makes it visible

Under streaming, the server must buffer before emitting. If your stop sequence is </answer> and the model has just produced </ans, the server cannot forward those characters — they might complete the stop string.

So it holds them until it knows. This produces the small stutters you sometimes see in a streamed response, where output pauses briefly and then arrives in a burst. Longer stop sequences make it more pronounced, which is a mild argument for short delimiters.

On the client side, never parse partial stream content against your own delimiters and assume it will match what the server does. Buffer to the end of the message and use the finish reason. Streaming and server-sent events covers handling the stream properly.

Where they still earn their place

The classic use is separating a model's turn from what comes next in a scripted format. In a ReAct-style agent loop where the model writes a thought and an action and your code writes the observation, stopping on the observation marker prevents the model from hallucinating the result of its own tool call. Agent loop anatomy covers where that boundary sits.

They are also a cheap guard against runaway output on open-ended tasks. A stop on a section delimiter caps cost when a model decides to keep going, which is worth having on anything user-facing. Agent token budgets covers the wider version of that control.

What they are not is a way to get structured output. Coaxing JSON by stopping on a closing brace fails the moment the object contains a nested one. Use a grammar constraint or a native tool-calling interface, both of which guarantee validity rather than approximating it. Structured outputs and JSON mode covers the options, and tool calling explained covers when the provider handles this for you.

The takeaway

Three rules cover almost every stop-sequence bug. Check the finish reason on every call and treat a length stop as a failure. Assume the stop text is stripped from your output unless you have verified otherwise for your provider. And pick delimiters the model has no reason to produce spontaneously — an XML-style tag rather than a bare word.

If you find yourself building elaborate stop logic, that is a sign the format is doing work the API should be doing. Structured output and tool calling exist for exactly that reason.