Handling SSE Disconnects Without Losing the Response

A streamed completion that dies halfway through does not raise an exception. The status line was 200 before the first token left the model, the headers are long gone, and everything after that is just bytes on a socket that stopped arriving. Your HTTP client has no opinion about it.

That is the whole problem with server-sent events in front of an LLM. The failure mode is not an error you catch, it is an absence you have to notice. Code written against the ordinary request-response shape treats a half-finished answer as a finished one, and then parses it, stores it, or shows it to a user.

Know what a complete stream looks like

An OpenAI-compatible SSE stream ends in a specific way, and you should assert on it rather than assuming it. The final content chunk carries a non-null finish_reason, and the transport then sends a literal data: [DONE] sentinel before closing.

Both signals matter and they mean different things. The finish reason tells you why generation stopped — stop for a natural ending, length for hitting the output cap, tool_calls when the model wants to call a function. The sentinel tells you the transport agreed the stream was over.

If the socket closes and you never saw a finish reason, you have a truncated response. Treat it as a failed request, not a short one. The same discipline applies to the non-streaming path, as covered in the walkthrough of the errors an LLM API actually returns, but streaming is where people skip it most often.

saw_finish = False
for event in stream:
    if event.data == "[DONE]":
        break
    chunk = json.loads(event.data)
    choice = chunk["choices"][0]
    if choice.get("finish_reason"):
        saw_finish = True
    delta = choice["delta"].get("content")
    if delta:
        yield delta

if not saw_finish:
    raise TruncatedStream(len(collected))

Idle timeouts, not total timeouts

A total request timeout is the wrong instrument for a stream. A long generation legitimately takes minutes; a dead stream produces nothing for minutes too. One number cannot separate them.

What you want is an idle timeout: the maximum gap between two chunks. Thirty seconds of silence in the middle of a generation is pathological even when the whole call is allowed to run for ten minutes. Most HTTP clients express this as a read timeout, which is applied per socket read rather than to the whole request, so it does exactly the right thing here.

Set the first-token budget separately and make it tighter. Time to first token is dominated by queueing and prefill, so a stream that has not produced a single byte in fifteen seconds is usually queued behind capacity you are not going to get. The reasoning behind those numbers is worked through in the guide to picking timeouts for LLM calls.

Heartbeats keep the middle boxes honest

Every hop between you and the model has its own idea of how long a quiet connection may live. Load balancers, reverse proxies and cloud ingress controllers commonly close idle connections after somewhere between thirty and sixty seconds, and none of them will tell your application why.

The standard fix is a comment frame — a line starting with a colon — sent periodically when there is nothing else to send. It is valid SSE, clients ignore it, and it resets every idle timer on the path.

: keepalive

data: {"choices":[{"delta":{"content":"hello"}}]}

If you are relaying a provider stream through your own service, you have to emit these yourself during the prefill gap, because the upstream may be silent for the entire time the model is reading a long prompt. The other half of that job — buffering, header handling and flushing — is covered in the notes on running your own LLM proxy.

Buffering is what breaks streaming in production

The single most common report of "streaming works locally but not deployed" is a proxy that accumulates the response body before forwarding it. The stream still completes, it just arrives all at once at the end, which defeats the point.

Three things have to be true on every hop. Response buffering must be off — in nginx that means proxy_buffering off or the X-Accel-Buffering: no response header. Compression must not be applied to the stream, because a compressor will happily wait for a block boundary. And the connection must be HTTP/1.1 or later with chunked transfer encoding intact.

Test this against the deployed path, never against localhost. Curl with --no-buffer is enough to see whether tokens trickle or arrive as one lump.

Retrying a stream is not free

When a stream dies at token 900 of a planned 1,200, you have already paid for the input tokens and for the output generated so far. Restarting the request pays for all of it again, and there is no resume protocol — SSE has a Last-Event-ID mechanism, but inference providers do not offer replay of a completion you were halfway through.

So the decision is economic as much as technical. For a short answer, retry from scratch. For a long generation on an expensive model, it is often better to surface the partial output and let the caller decide, or to re-prompt with the partial text as an assistant prefix so the model continues rather than restarts.

Whichever you choose, bound it. One retry on a truncated stream is reasonable; a retry loop on a provider that is dropping every connection is an amplifier, and the reasons are the same ones described in the mechanics of exponential backoff.

Client disconnects should cancel upstream work

Streaming makes abandonment common. Users close tabs, navigate away, and hit stop. If your server keeps reading from the provider after the browser has gone, you are paying for tokens nobody will ever read and holding a connection slot that a live request needs.

Propagate cancellation. In Node, listen for the request close event and abort the upstream fetch through an AbortController. In Python, catch the client-disconnect signal your framework raises and close the response context. This is cheap to implement and it frees real capacity under load, which matters more than any retry tuning once you are near your quota.

What to instrument

Four numbers make stream failures diagnosable rather than mysterious. Count streams that ended without a finish reason. Record time to first token separately from total duration. Record the largest inter-chunk gap per stream. And record the finish reason distribution, because a rise in length means your output cap is too low, not that the provider is unhealthy.

Those metrics fit naturally alongside the request-level spans described in the piece on tracing agent runs. Without them, every truncation report becomes a manual reproduction attempt against a non-deterministic system.

A working checklist

1. Assert on finish_reason and the [DONE] sentinel before treating output as complete.
2. Use an idle read timeout plus a separate, tighter first-token timeout.
3. Emit heartbeat comment frames during silent periods.
4. Disable buffering and compression on every proxy hop, then verify on the deployed URL.
5. Cancel upstream requests when the client disconnects.
6. Retry truncated streams at most once, and decide deliberately between restart and continuation.