Timeout Tuning for LLM Calls: Picking Numbers That Hold

Most services inherit their LLM timeout from whatever the SDK ships with, discover it during an incident, and then set it to a number someone guessed in a meeting. Both failure modes are avoidable, because the right numbers are derivable from latency data you already have.

The first thing to fix is the assumption that there is one timeout. There are at least four, they answer different questions, and collapsing them into a single value is why the setting never feels right.

Four timeouts, four jobs

The connect timeout bounds establishing the TCP and TLS connection. It should be short — a couple of seconds — because a healthy provider connects in tens of milliseconds and anything slower is a network or DNS problem that will not improve by waiting.

The first-token timeout bounds how long you wait for the model to start producing. This is dominated by queueing and prefill, so it scales with prompt length and with how loaded the provider is. It is the timeout that catches capacity problems.

The idle timeout bounds the gap between consecutive chunks in a stream. It is the only reliable way to distinguish a long generation from a dead connection, and it belongs on every streaming call.

The overall deadline bounds the whole operation including retries. It exists to protect the caller, not the call, and it is the one that should be derived from a product requirement rather than from provider behaviour.

Derive the numbers, do not guess them

Set every timeout except the overall deadline from your own measured distribution for that model and prompt shape. A workable rule is to take the 99th percentile of successful calls and multiply by two to three.

Tighter than that and you will time out requests that would have succeeded, converting a slow response into a failure and, if you retry, into two charges. Looser and the timeout stops doing anything: a value that no failing request ever reaches is decoration.

Measure per model, not globally. A small fast model and a large reasoning model have latency distributions that differ by an order of magnitude, and a single number tuned for one is wrong for the other. The factors that drive those differences are broken down in what actually determines inference latency.

Re-measure after any change to the prompt, the model version or the output cap. Prefill time scales with input length, so a context-stuffing change can double your first-token latency without touching the model.

The default is usually far too long

Provider SDKs ship with generous defaults, in the range of several minutes, because they must accommodate the slowest legitimate workload anyone runs. The OpenAI Python client defaults to ten minutes per request.

Inside a web request handler, a ten-minute timeout is equivalent to no timeout. Your connection pool exhausts, your workers pile up, and the user gave up nine minutes earlier. Set it explicitly on the client, and set it per call site rather than once globally, because a background batch job and an interactive endpoint want different values.

fast = OpenAI(timeout=httpx.Timeout(
    connect=2.0, read=20.0, write=10.0, pool=5.0
), max_retries=0)

batch = fast.with_options(timeout=httpx.Timeout(
    connect=2.0, read=180.0, write=10.0, pool=5.0
))

Note max_retries=0. Timeouts and retries multiply, so if the SDK retries twice inside a thirty-second read timeout you have committed to ninety seconds, not thirty. Own retries in one layer, as argued in the guide to exponential backoff.

Read timeouts are per-read, and that is the point

The common misunderstanding is that a read timeout bounds the whole response. In most HTTP clients it bounds a single socket read — the maximum time with no bytes arriving.

For a non-streaming call that is effectively a total timeout, because nothing arrives until the response is complete. For a streaming call it is exactly the idle timeout you want, and it will happily allow a ten-minute generation as long as tokens keep flowing.

That asymmetry means the same configured value behaves completely differently depending on whether you set stream=true, which surprises people who tuned it on one path and deployed it on the other. If you need a hard ceiling on a stream, enforce it yourself with a deadline check in the consumption loop, and handle the partial output deliberately — the options are covered in handling SSE disconnects.

Propagate a deadline instead of nesting timeouts

Independent timeouts at each layer produce an incoherent total. An inner call with a thirty-second timeout inside a handler with a twenty-second budget will keep working long after the response has been abandoned, holding a slot and paying for tokens nobody reads.

Compute one deadline at the edge and pass it down. Each layer sets its own timeout to the minimum of its configured value and the time remaining, and refuses to start work when the remaining time is less than a plausible attempt.

This also gives retries a natural stopping condition. Rather than counting attempts, you stop when the deadline cannot accommodate another one, which is the behaviour you actually wanted from a retry cap.

Cancel the work, not just the wait

A timeout that only stops your thread from waiting still leaves the provider generating and billing. Attach a cancellation token to every call and fire it when the deadline passes or the client disconnects.

In Node that is an AbortController wired to both the timer and the incoming request close event. In Python it is closing the response context or cancelling the task. Either way, verify it actually aborts the underlying socket, because a wrapper that merely raises in your code while the transfer continues in the background gives you the latency benefit and none of the cost benefit.

Under load this is not a minor optimisation. Abandoned streams tie up concurrency slots you are paying for and can push you into rate limits that look like a provider problem — the accounting is examined in why agent costs are unpredictable.

Different call sites deserve different budgets

An autocomplete suggestion, a chat turn, and an overnight batch classification have nothing in common except the API they call. Give them separate clients with separate timeouts and separate concurrency limits.

A reasonable starting point: interactive completions get a first-token budget in the low tens of seconds and a total deadline tied to what the interface can tolerate. Background jobs get minutes and a queue. Agent tool loops get a per-step timeout plus a whole-run deadline, because a single step timing out should not necessarily kill a run that is otherwise progressing, a distinction developed in running agents that stay alive for hours.

A short procedure

1. Instrument time to first token and total duration per model.
2. Set connect to about two seconds everywhere.
3. Set first-token and idle timeouts to two or three times the observed p99.
4. Set the overall deadline from the product requirement, not the provider.
5. Disable SDK retries so timeouts do not multiply.
6. Propagate one deadline through every layer.
7. Cancel upstream work on timeout and on client disconnect.

Then alert on the ratio of timeouts to successes rather than on absolute counts. A timeout rate that climbs while latency is flat usually means load, and a timeout rate that climbs together with p99 usually means the provider — and knowing which one before you start debugging is most of the value.