Proxying LLM Traffic: What a Gateway Has to Get Right

Every organisation past its second LLM integration ends up wanting a gateway. The reasons are always the same: one place to hold provider keys, one place to attribute spend, one place to switch models without redeploying six services.

The reasons are good. The implementation is where it goes wrong, because an LLM proxy is not a normal reverse proxy — it forwards long-lived streaming responses, and the default behaviour of nearly every HTTP framework is to buffer those.

What a gateway buys you

Key custody is the first and largest benefit. With a gateway, application code holds an internal credential and never sees a provider key, so rotating a provider key is one deployment rather than a scavenger hunt across repositories and CI secrets. The multi-tool version of that problem is described in using one API key across tools.

Attribution is the second. A gateway sees every request, so it can record tokens, model and latency per team, per service and per feature — the numbers you need before you can answer any question about spend, as laid out in token accounting for finance.

Control is the third: model routing, fallbacks, per-tenant rate limits and budget caps all become configuration rather than code in each client. The routing side is covered in model routing and fallbacks.

The cost is a hop in the critical path and one more thing that can be down. That trade is usually worth it above a few services and rarely worth it below two.

Streaming is the part that breaks

Frameworks buffer response bodies by default. So do most reverse proxies. So does gzip. Any one of those turns a streamed completion into a single delivery at the end, and the client sees a working request with terrible time to first token.

Three settings have to be right on every hop. Response buffering off — in nginx that is proxy_buffering off, or the X-Accel-Buffering: no response header from your application. No compression on the SSE response. And chunked transfer encoding preserved, which means not setting a Content-Length and not accidentally downgrading to HTTP/1.0.

location /v1/ {
    proxy_pass https://upstream/v1/;
    proxy_buffering off;
    proxy_request_buffering off;
    proxy_http_version 1.1;
    proxy_read_timeout 600s;
    gzip off;
}

Test on the deployed URL rather than locally. Local runs usually skip the ingress controller and the CDN, which are exactly the hops that buffer. A curl with --no-buffer that shows tokens trickling is the only acceptable evidence.

Forward the right headers, and only those

Strip the client credential and inject the provider one. That is the whole point, and it means your gateway must never pass through an inbound Authorization header by accident.

Forward what the provider needs and drop the rest. Hop-by-hop headers — Connection, Keep-Alive, Transfer-Encoding, Upgrade — must not be relayed. Content-Length must be recomputed if you touch the body at all, and stripped entirely for streamed responses.

In the other direction, relay the rate-limit headers back to callers. The remaining-requests and remaining-tokens counters are how client teams learn they are approaching a limit, and swallowing them at the gateway leaves everyone blind — the reasons are in pacing under rate limits.

Cancellation must propagate both ways

When a caller disconnects mid-stream, the gateway has to abort its upstream request. If it does not, you keep paying for a generation nobody will read and you hold a connection slot for its full duration.

This is easy to get wrong because the naive pipe implementation only notices the client is gone when it next tries to write, which may be many seconds later. Listen for the connection close event explicitly and cancel the upstream call from it.

The reverse direction matters too. When the upstream stream dies without a finish reason, the gateway should surface that as an error rather than closing the client stream cleanly, or every consumer will treat a truncated answer as complete. The detection technique is in handling SSE disconnects.

Do not become a bottleneck

A gateway concentrates all your inference concurrency into one process pool. Sizing it like an ordinary API service is how it falls over.

Use an async runtime, because thread-per-request models cap you at the thread count and LLM requests are long. Size the outbound connection pool for concurrent in-flight requests rather than request rate — the arithmetic is in sizing HTTP pools for LLM APIs. And run more than one replica, since a single-instance gateway makes provider redundancy pointless.

Keep the hot path thin. Anything expensive — usage aggregation, log shipping, evaluation sampling — belongs on a queue behind the response, not inline. A gateway that adds fifteen milliseconds is invisible next to a two-second completion; one that adds two hundred while it writes to a database is not.

Model aliasing is the feature people underrate

Let clients ask for a stable alias and resolve it to a concrete model at the gateway. Clients request something like default-coder, and the gateway maps it to a specific versioned model.

That gives you a migration path that does not require touching client code, and it gives you a canary mechanism: route a percentage of traffic to a new model, compare outcomes, then move the alias. It also stops a hundred services from hardcoding a model string that will eventually be retired.

The discipline that makes it safe is recording the resolved model on every response and in every log line. Otherwise a change in what the alias points at becomes an unattributable quality regression, which is the argument made in pinning model versions.

Logging without collecting a liability

A gateway sees every prompt in the organisation, which makes it the single most sensitive log source you operate and the most tempting place to log everything.

Default to metadata: model, token counts, latency, status, caller identity, a content hash. Store bodies only when a caller opts in, only for a short retention window, and only through the same redaction pipeline you would apply anywhere else — the specifics are in logging LLM requests safely.

Be explicit about it in the gateway’s own documentation. Teams will assume the gateway logs prompts unless told otherwise, and that assumption changes what they are willing to send through it.

Before you build one

A gateway is a real service with real operational weight, and there are mature open-source implementations that already handle streaming, routing and usage accounting. Building your own is justified when you need routing logic or tenancy rules that off-the-shelf options do not express.

If you do build, the acceptance tests are specific: tokens stream through the deployed path without buffering, a client disconnect aborts the upstream request within a second, a truncated upstream stream surfaces as an error, rate-limit headers reach the caller, and no provider key appears in any log. Those five cover most of what goes wrong.