Streaming in Python: Reading SSE From an LLM API

Streaming exists for one reason: a user who sees the first word in 400 milliseconds tolerates a response that takes twenty seconds, and a user who sees nothing for twenty seconds assumes the thing is broken. Every complication below follows from wanting that first token early, and every mistake below is a way of accidentally giving it up.

The SDK will do all of this for you, and for most application code it should. This article is about what happens underneath, because when a stream stalls, truncates or arrives in one lump, the SDK abstraction is exactly what stops you from seeing why.

The wire is simpler than it looks

Send the ordinary chat completions request with "stream": true. The response comes back with a content type of text/event-stream and stays open, with the server writing frames as tokens are generated.

Each frame is a line beginning with data: followed by a JSON object, then a blank line as a separator. The JSON has the same overall shape as a non-streaming response, except that choices[0] carries a delta rather than a message, and the delta holds only what is new since the last frame — usually a handful of characters.

The stream terminates with a sentinel frame whose payload is the literal text [DONE] rather than JSON. Passing that to a JSON parser is the first bug almost everyone writes, and it surfaces as a decode error at the very end of an otherwise perfect response. Check for the sentinel before parsing, always.

Not every frame carries text. The first typically has a role and no content, some carry only a finish reason, and many providers emit periodic comment lines beginning with a colon as keepalives. Ignore anything that is not a data line and tolerate deltas with nothing in them. The protocol-level detail is in the server-sent events explainer.

Reading it with httpx

The minimum viable consumer. Note that the request is made inside a context manager so the connection is closed even if the caller stops early, and that the stream is iterated line by line rather than read in full.

import json, httpx

def stream(prompt, key, base_url, model):
    body = {
        "model": model,
        "stream": True,
        "messages": [{"role": "user", "content": prompt}],
    }
    headers = {"Authorization": "Bearer " + key}
    timeout = httpx.Timeout(connect=5.0, read=30.0, write=10.0, pool=5.0)

    with httpx.Client(timeout=timeout) as client:
        with client.stream("POST", base_url + "/chat/completions",
                           json=body, headers=headers) as r:
            r.raise_for_status()
            for line in r.iter_lines():
                if not line or line.startswith(":"):
                    continue
                if not line.startswith("data:"):
                    continue
                payload = line[5:].strip()
                if payload == "[DONE]":
                    return
                chunk = json.loads(payload)
                delta = chunk["choices"][0].get("delta", {})
                text = delta.get("content")
                if text:
                    yield text

The requests equivalent is iter_lines on a response opened with stream=True, and it works, but note that a bare timeout in requests applies between reads rather than to the whole response — a subtlety that bites when a generation stalls mid-stream.

Partial chunks, and why iter_lines is not always enough

TCP delivers bytes, not lines. A single data frame can arrive split across two network reads, and two frames can arrive in one. Library helpers like iter_lines handle the reassembly for you, which is exactly why you should use them rather than iterating raw chunks.

If you do work at the byte level — because you are proxying, or because you need the raw frames for logging — you must maintain a buffer. Append each chunk, split on the newline boundary, process every complete line, and keep the remainder for the next iteration. Dropping that remainder produces the most confusing bug in this whole area: output that is correct most of the time and silently loses a few characters under load, because the split only lands mid-frame when packets are large.

The related trap is decoding. Split the bytes into lines first, then decode each line as UTF-8. Decoding an arbitrary chunk boundary can slice a multi-byte character in half, which throws on some inputs and produces replacement characters on others — and the inputs that trigger it are non-English text and emoji, so it reliably escapes English-only testing.

Accumulating tool-call deltas

Tool calls stream too, and they stream worse than text. Rather than a complete call in one frame, you get a tool_calls array inside the delta where each element has an index, and the function name arrives in one frame while the arguments arrive as a JSON string built up a few characters at a time across many frames.

So you cannot parse the arguments until the stream ends. Accumulate by index into a dictionary, concatenating the argument fragments, and only call json.loads on the assembled string once the stream completes.

calls = {}
for chunk in frames:
    for tc in chunk["choices"][0].get("delta", {}).get("tool_calls", []):
        slot = calls.setdefault(tc["index"], {"id": "", "name": "", "args": ""})
        if tc.get("id"):
            slot["id"] = tc["id"]
        fn = tc.get("function", {})
        if fn.get("name"):
            slot["name"] = fn["name"]
        slot["args"] += fn.get("arguments", "")

Two things follow. Never try to parse partial arguments as JSON hoping for an early start — it will succeed occasionally by coincidence and produce a truncated object. And check the finish reason before dispatching: a stream that ended because it hit the token limit leaves you holding half an argument string, and executing a tool with truncated arguments is worse than not executing it. The tool calling walkthrough covers the non-streaming shape, and function calling in Python covers the dispatch side.

Timeouts and cancellation

A streaming request needs different timeouts from a normal one. A total-request timeout is close to useless here, because a long generation legitimately takes a long time. What you want is a read timeout: the maximum gap between frames. If nothing has arrived for thirty seconds, the generation has stalled or the connection has died silently, and either way you should give up.

That distinction is why the example above sets connect and read timeouts separately rather than a single number. Getting it wrong in the other direction is worse than it sounds: too short a read timeout kills healthy long generations, and the resulting retry costs you the full price of the abandoned output. Tuning timeouts works through picking values, and handling SSE disconnects covers what to do when one dies halfway through.

Cancellation matters for cost. If the user closes the tab or your consumer breaks out of the loop, close the response so the connection tears down — a generator abandoned without cleanup can leave the server generating tokens you will be billed for and will never see. Using the context manager form makes this automatic; assigning the response to a variable and forgetting it does not. In async code, a cancelled task should still close the stream in a finally block.

Buffering defeats the entire point

Everything above can be correct and the user can still see nothing for twenty seconds, because something between the model and the screen accumulated the output before passing it on.

The usual culprits, roughly in order. A reverse proxy with response buffering enabled, which will hold a whole event stream by default in some configurations. Your own web framework, if the handler builds a string and returns it rather than yielding. Compression middleware, which cannot flush a partial compressed block. And print statements to a pipe, where Python line-buffers to a terminal but block-buffers to anything else, so a script that streams beautifully interactively shows nothing when piped into another command.

Diagnose it from the outside in. Run the same request through curl and watch whether tokens appear progressively; if they do, the provider and the network are fine and the problem is inside your stack. Then remove one layer at a time. It is worth knowing that streaming does not change what you are billed — the token accounting is identical, as the cost of streaming sets out — so this is purely a perceived-latency problem, which is also why it is easy to leave broken for months.

When to just use the SDK

For application code, use the official client. It handles line reassembly, the sentinel, keepalives, retries on connect and the delta accumulation, and it will keep handling them as providers add fields. Writing it yourself is a way to learn the format, not a way to save work; setting up the Python SDK against a custom endpoint takes two lines.

Write the raw version when you are proxying traffic and need to pass frames through untouched, when you are logging exactly what arrived for debugging, or when you are diagnosing a stream that the SDK reports as simply empty. Those are real cases, and they are why the format is worth understanding even if you never ship the code.

The takeaway

Check for the sentinel before parsing. Use a line-oriented reader, and if you cannot, keep a remainder buffer and decode after splitting. Accumulate tool-call arguments by index and parse only at the end, after checking the finish reason. Set a read timeout rather than a total timeout, and close the response on cancellation. Then check for buffering in your proxy, your framework and your own stdout before believing the stream is broken.