Throughput vs Latency: The Trade-off Behind Your Bill

Two teams complain about the same inference deployment. One says it is slow. One says it is expensive. Both are right, and the reason is that the fix for either makes the other worse.

Throughput is total tokens the system produces per second across all users. Latency is how long one user waits. They are not the same measurement, they are not improved by the same changes, and almost every serving decision is a choice between them.

Why the two pull apart

During generation, the expensive operation is not arithmetic — it is moving model weights out of memory and into the compute units. Every decode step reads the weights, does a small amount of work per token, and reads them again for the next step.

If you process one request at a time, you pay that full memory traffic to produce a single token. If you process thirty-two requests together, you pay roughly the same memory traffic and produce thirty-two tokens. The weights were read once and used thirty-two times.

That is the whole economics of batching, and it explains why the same hardware can look either fast or cheap. Larger batches spread a fixed memory cost over more output, which is why throughput rises almost linearly with batch size long before compute saturates.

The cost is that each individual request now waits behind the batch. Its tokens arrive in step with everyone else, at whatever pace the batch runs. The mechanics of prefill and decode explain where that waiting actually lands.

The two latency numbers people conflate

Time to first token measures how long before anything appears. It is dominated by queueing and by prefill — processing your prompt. A long prompt or a busy queue both show up here, and users interpret it as the system being unresponsive.

Time per output token measures how quickly text continues once it starts. It is dominated by decode speed, which is where batch size bites. A batch that is too large makes generation visibly stutter even though the first token arrived promptly.

These two respond to different fixes. Cutting your prompt improves the first and does nothing for the second. Reducing concurrency improves the second and may make the first worse if requests start queueing instead.

Where throughput stops paying

Batch size does not improve throughput indefinitely. At some point the system stops being limited by memory bandwidth and starts being limited by compute or by memory capacity, and beyond that point every extra request in the batch slows everyone without producing more aggregate tokens.

Capacity usually binds first in practice. Each concurrent request holds a key-value cache that grows with its context length, and that cache occupies memory for the whole session. A deployment sized for the weights alone will run out of room for concurrency long before it runs out of compute, which is the single most common self-hosting mistake. What the KV cache actually stores covers why.

The practical shape is a curve with a knee. Throughput climbs steeply, flattens, and then latency degrades sharply for no further gain. Finding that knee on your own traffic is worth more than any published benchmark, because it depends on your context lengths.

What this means when you are buying tokens

If you call a hosted API, you are buying a position on somebody else's curve. The provider chose a batch size that maximises tokens per unit of hardware, because that is what their per-token price has to cover. You get the cost benefit and inherit the latency.

This is why a model can feel slower on one provider than another at identical prices, and why the same model feels faster at three in the morning. You are seeing queue depth and batch composition, not a different model.

It also explains the existence of premium low-latency tiers. Serving at small batch sizes costs more per token because the fixed memory traffic is spread thinner, so anyone offering guaranteed fast decode is charging for the throughput they gave up.

Deciding which one you are optimising

Interactive work — a chat interface, an editor completion, anything a person is watching — is latency-bound. A user notices a stall of a second. They do not notice that the fleet is running at sixty percent utilisation.

Batch work — nightly test generation, bulk classification, indexing a repository — is throughput-bound. Nobody is watching, so the only figures that matter are total tokens and total cost. Running these jobs with latency-optimised settings wastes money for no benefit anyone perceives.

The mistake is running both through one path. A bulk job dropped into the interactive queue inflates batch sizes and degrades the experience for everyone typing. Separate them, even if it means two deployments or two API keys, and batching the offline work deliberately.

What to measure

Measure at the percentile your users actually experience. Average latency hides the queueing that causes complaints; the ninety-fifth and ninety-ninth percentiles are where a batching policy reveals itself.

Measure under realistic concurrency. A single-request benchmark tells you the best case the hardware can produce and nothing about behaviour when twenty sessions share it. Load-test with your own prompt lengths, because context length drives cache occupancy and therefore the concurrency ceiling.

Record tokens per second per user alongside tokens per second in total. Reporting only the aggregate makes a badly latency-degraded system look healthy, and it is the number vendors prefer to quote.

A decision rule

Ask who is waiting. If a person is waiting, optimise time to first token first and decode speed second, and accept lower utilisation as the price. If a job queue is waiting, push batch size until the throughput curve flattens and ignore per-request latency entirely.

If both are true, split the traffic rather than compromising. A single middle setting is usually worse for both audiences than two committed settings. Choosing for interactive speed and weighing self-hosting against managed pricing both come down to which side of this trade-off you are standing on.