Connection Pooling for LLM APIs: Sizing the Client Right

HTTP client defaults were designed for requests that finish in tens of milliseconds. An LLM call holds a connection open for seconds or minutes, which turns two harmless defaults — a small pool and an unbounded queue — into a latency problem that looks like the provider being slow.

The symptom is distinctive: your measured request duration climbs while the provider reports normal latency, and the gap grows with concurrency. That gap is time spent waiting for a connection, and it is entirely on your side.

Create the client once

The most expensive mistake is also the easiest to make. Constructing a new SDK client inside a request handler creates a new connection pool, which means a fresh TCP handshake and a fresh TLS negotiation on every call.

On a cross-region connection that is a round trip for TCP and one or two more for TLS, so somewhere in the low hundreds of milliseconds added to every request, plus the CPU cost of the handshake. It also defeats HTTP/2 multiplexing entirely, because you never reuse a connection long enough to benefit.

Make the client a module-level singleton, or bind it into your dependency container at startup. In serverless runtimes, construct it outside the handler function so it survives across warm invocations. Then use per-call option overrides rather than new clients when a specific call needs a different timeout.

Pool size is a function of concurrency, not traffic

The number of connections you need is the number of requests in flight at once, not requests per second. For short calls those numbers are similar; for LLM calls they are not.

The arithmetic is Little’s law. A hundred requests per second at fifty milliseconds each needs about five concurrent connections. A hundred requests per second at twenty seconds each needs two thousand. Most default pools are sized somewhere between ten and a hundred, which is fine for the first case and catastrophic for the second.

So compute it from your own numbers: peak concurrent requests equals arrival rate times mean duration, then add headroom. If that number is uncomfortably large, the fix is usually a concurrency limit rather than a bigger pool, because an unbounded pool just moves the queue to the provider.

limits = httpx.Limits(
    max_connections=200,
    max_keepalive_connections=200,
    keepalive_expiry=90.0,
)
client = OpenAI(http_client=httpx.Client(limits=limits, timeout=...))

Pool starvation looks like provider latency

When the pool is full, new requests wait. Depending on the client that wait is either bounded by a pool timeout or unbounded, and unbounded is the default in more libraries than you would expect.

Set an explicit pool acquisition timeout, and make it short. Waiting five seconds for a connection before starting a twenty-second call is almost never better than failing fast and shedding the request, and a fast rejection is a signal you can act on rather than a mysterious tail latency.

Then measure the wait. Most clients expose pool statistics; if yours does not, wrap acquisition and time it. A histogram of connection acquisition time is the single most useful metric for separating your own saturation from an upstream slowdown, which is otherwise a guessing game — the broader instrumentation story is in tracing LLM and agent calls.

Keepalive and the idle-close race

Keepalive is what makes pooling worthwhile, and it has a specific failure mode. If the server closes an idle connection at the same moment your client picks it up, you get a connection reset that looks random and clusters at low traffic.

The defence is to expire connections on your side before the server does. Set the keepalive expiry below the provider or intermediary idle timeout — sixty to ninety seconds is usually safe — so the client discards stale sockets rather than racing on them.

Also make sure a connection error on a fresh socket is retried once, since a reset before any bytes were sent is unambiguously safe to repeat. That is one of the few retries that needs no idempotency argument, and it belongs in the classification described in the backoff guide.

Streaming changes the accounting

A streamed completion holds its connection for the entire generation, which means concurrency is bounded by pool size in a way that is easy to miss when you are looking at request rate.

Abandoned streams make it worse. When a user closes a tab and your code keeps reading, the connection stays occupied for the full generation, so a small amount of abandonment can consume a surprising fraction of the pool. Cancelling upstream on client disconnect frees those slots immediately — the mechanics are in handling SSE disconnects.

If you run a mix of interactive and batch traffic, give them separate clients with separate pools. Sharing one pool means a batch job can starve interactive requests, and no amount of retry tuning fixes a queue you are stuck behind.

HTTP/2 helps, until it does not

Under HTTP/2 many requests multiplex over one connection, so pool pressure largely disappears and the relevant limit becomes the peer’s maximum concurrent streams rather than your connection count.

That is usually a win. The caveat is that everything now shares one TCP connection, so packet loss stalls all streams on it, and a single connection can become a throughput bottleneck for large payloads. If you send very long prompts at high concurrency, forcing a small number of parallel connections often beats one multiplexed one.

Check what you actually negotiated rather than assuming. Proxies and load balancers frequently downgrade to HTTP/1.1 without saying anything, and that changes your pool sizing completely. If you route through your own gateway, the hop-by-hop details matter — see running your own LLM proxy.

Runtime-specific traps

In Node, the global fetch agent and the older http agent have different defaults, and a library may use either. Set an explicit agent with your chosen maxSockets and keepAlive rather than trusting whichever one is in play.

In Python, a synchronous client in a thread pool caps concurrency at the number of threads, whatever the pool says. The async client removes that ceiling and immediately exposes whether your pool limits were doing any work.

In serverless environments, pooling helps only across warm invocations of the same instance, and a cold start pays every handshake again. That is one reason a shared long-lived gateway in front of a fleet of short-lived functions often outperforms letting each function talk to the provider directly.

What to check first

When request latency exceeds provider-reported latency, work through four things in order. Confirm the client is a singleton. Confirm the pool is sized for concurrent requests, not request rate. Confirm the pool acquisition timeout is set and short. Confirm keepalive expiry is below the upstream idle timeout.

Those four cover the large majority of self-inflicted latency on LLM calls, and all four are configuration rather than architecture. Only after they are settled is it worth looking at the provider, a fallback route, or a bigger machine.