Streaming an OpenAI-Compatible API in TypeScript

The SDK hides the stream behind an async iterator and most of the time that is the right choice. You end up writing this by hand when you need to proxy the bytes somewhere else, splice in your own events, or run inside a runtime the SDK does not target. It is not hard, but three details reliably break naive implementations: chunk boundaries, tool-call deltas, and cancellation.

What follows is the mechanics of consuming a Server-Sent Events response from an OpenAI-compatible endpoint using nothing but fetch and the Web Streams API. The protocol itself is covered in streaming and server-sent events; this is the TypeScript that sits on top of it.

What actually arrives on the wire

Set stream: true and the response body is a sequence of text frames rather than one JSON document. Each frame is a line beginning with data: , followed by a JSON payload, followed by a blank line. The stream ends with a sentinel frame whose payload is the literal string [DONE] rather than JSON.

Each payload has the same shape as a non-streaming completion except that the interesting content lives under choices[0].delta instead of choices[0].message. A delta carries whichever fields changed: usually a fragment of content, sometimes a role on the first frame, sometimes a slice of a tool call, and on the final frame a finish_reason with an empty delta.

Fragments are not tokens and not words. They are whatever the server decided to flush, which depends on its batching and on the network path. Never write logic that assumes a delta ends on a word or a syntactic boundary. Some providers also emit comment lines starting with a colon as keep-alives, and those must be skipped rather than parsed.

Reading the body as a stream

The response body is a ReadableStream<Uint8Array>. You need a reader, a decoder that tolerates split multi-byte characters, and a loop.

const res = await fetch(url + '/chat/completions', {
  method: 'POST',
  headers: {
    'content-type': 'application/json',
    authorization: 'Bearer ' + apiKey,
  },
  body: JSON.stringify({ model, messages, stream: true }),
})

if (!res.ok || !res.body) {
  throw new Error('stream failed: ' + res.status + ' ' + (await res.text()))
}

const reader = res.body.getReader()
const decoder = new TextDecoder('utf-8')

The TextDecoder matters more than it looks. A multi-byte UTF-8 character can be split across two network chunks, and decoding each chunk independently produces a replacement character in the middle of otherwise fine output. Passing { stream: true } to decode makes the decoder hold the trailing incomplete bytes until the next call.

Check the status before you start reading. A rejected request still returns a body, but that body is a JSON error object, not SSE, and a parser expecting frames will produce a confusing failure instead of the real message. The catalogue of what you might get back is in the LLM API error code reference.

Buffering across chunk boundaries

The single most common bug is treating each network chunk as a complete frame. It is not. One chunk can contain six frames, half a frame, or a frame boundary in the middle of a JSON string. The fix is a string buffer that persists across iterations and is drained only on complete separators.

let buffer = ''
let text = ''

while (true) {
  const { done, value } = await reader.read()
  if (done) break
  buffer += decoder.decode(value, { stream: true })

  const frames = buffer.split('\n\n')
  buffer = frames.pop() ?? ''

  for (const frame of frames) {
    const line = frame.split('\n').find((l) => l.startsWith('data:'))
    if (!line) continue
    const payload = line.slice(5).trim()
    if (payload === '[DONE]') return text
    const delta = JSON.parse(payload).choices?.[0]?.delta
    if (delta?.content) text += delta.content
  }
}

The pattern to internalise is the pop. Splitting on the double newline gives you n complete frames plus one trailing remainder, which may be an empty string if the chunk happened to end on a boundary. Putting that remainder back into the buffer rather than parsing it is the whole trick.

Be slightly tolerant on the separator. Some proxies normalise line endings, so splitting on a regex that accepts an optional carriage return before each newline saves you a debugging session when a load balancer is in the path.

Accumulating tool-call deltas

Content deltas concatenate. Tool calls do not, and this is where most hand-rolled clients go wrong. A tool call arrives as a sequence of partial objects, each carrying an index, and the argument JSON is streamed as a string in fragments that are not individually valid JSON.

type Partial = { id?: string; name?: string; args: string }
const calls = new Map<number, Partial>()

for (const tc of delta.tool_calls ?? []) {
  const slot = calls.get(tc.index) ?? { args: '' }
  if (tc.id) slot.id = tc.id
  if (tc.function?.name) slot.name = tc.function.name
  if (tc.function?.arguments) slot.args += tc.function.arguments
  calls.set(tc.index, slot)
}

Key on index, never on array position and never on id, because parallel tool calls interleave and the id may only appear on the first fragment of each call. Only attempt JSON.parse on the accumulated argument string after the stream reports finish_reason of tool_calls. Parsing early throws on valid, incomplete input.

Once parsed, validate against the schema before dispatching. A truncated stream can leave you with arguments that parse but are missing required fields, and the typed dispatch pattern in function calling in TypeScript is the place to enforce that.

Cancellation with AbortController

Streams make abandonment expensive if you do not handle it. A user navigating away or an agent step that has already got what it needs should stop generation, not merely stop reading it, because tokens are billed as they are produced.

const ac = new AbortController()
const timer = setTimeout(() => ac.abort(), 120_000)

try {
  const res = await fetch(url, { signal: ac.signal, /* ... */ })
  // consume as above
} catch (err) {
  if ((err as Error).name !== 'AbortError') throw err
} finally {
  clearTimeout(timer)
  await reader.cancel().catch(() => {})
}

Pass the signal to fetch, not just to your own loop. Breaking out of the read loop without aborting leaves the connection open and the server generating. Calling reader.cancel() in a finally block covers the case where you exit early through a return or a thrown error.

Set the timeout on total stream duration and, separately, on the gap between chunks. A stream that delivers one token every ninety seconds is dead in every way that matters, and only an inactivity timer catches it. Timeout tuning for LLM calls covers choosing the numbers.

Backpressure and what actually stalls

Reading with getReader gives you natural backpressure: nothing is buffered beyond what you have read, so a slow consumer slows the transfer rather than growing memory. That property disappears the moment you do heavy synchronous work inside the loop.

The realistic failure is not memory but latency. Parsing every frame, updating React state per fragment, and writing to a database on each delta all add up when frames arrive a few milliseconds apart. Buffer the render: accumulate into a string and flush on an animation frame or a short interval rather than on every delta.

If you are re-emitting the stream to your own clients, use pipeThrough with a TransformStream rather than reading into memory and re-sending. That preserves the backpressure chain end to end and keeps your proxy from becoming the buffer, which is the shape described in proxying LLM traffic.

Browser and Node differences that bite

Modern Node has fetch, ReadableStream, TextDecoder and AbortController as globals, so the code above runs unchanged in both. The differences are around it rather than in it.

In the browser you cannot set arbitrary headers on EventSource, which is why everyone uses fetch instead — and it means your API key would be exposed to the page, so browser streaming should terminate at your own server rather than at the provider. Node has no such constraint but does have its own: an unhandled abort can surface as an ECONNRESET from the underlying socket rather than a clean AbortError, so match on both.

Edge and serverless runtimes add a third case. Many enforce a maximum response duration that is shorter than a long generation, and some buffer responses unless you explicitly stream, which silently converts your SSE endpoint into a slow non-streaming one. Test on the real runtime, not locally. What to do when the connection drops mid-generation is covered in handling SSE disconnects.

A short checklist

Before shipping a hand-rolled stream client, confirm six things. The decoder is called with { stream: true }. The frame buffer keeps its remainder. The [DONE] sentinel is checked before JSON.parse. Tool-call fragments accumulate by index and are parsed once at the end. An AbortController signal is attached to the fetch itself. And an inactivity timeout exists, not only a total one.

Then test the ugly cases deliberately: kill the connection mid-stream, send a prompt that triggers two parallel tool calls, and stream text containing multi-byte characters. Those three cover almost every bug this code has.