Batch Size and Throughput: The Trade Behind Every Token Price

Serving one request at a time on a modern accelerator wastes most of it. Batching is what makes inference economically viable, and it is also the reason your latency changes depending on what everyone else is doing.

You do not configure batching when you call an API — the provider does — but understanding it explains the pricing, the latency variance and several behaviours that otherwise look like the provider misbehaving.

Why a single request wastes the hardware

Generating one token means reading the model weights from memory and doing a comparatively small amount of arithmetic with them. The bottleneck is the reading, not the arithmetic.

So during single-request decode, memory bandwidth saturates while the arithmetic units sit largely idle. The expensive part of the machine is not the part doing the work.

Batching fixes the mismatch. If eight sequences generate their next token together, the weights are read once and used eight times. Memory traffic is roughly unchanged; useful output multiplies. That is the whole idea, and it is why throughput can improve enormously with almost no additional memory traffic.

Prefill behaves differently. Processing a prompt is already compute-bound and already parallel across tokens, so a long prompt saturates the arithmetic units on its own. LLM inference latency explained covers the two phases.

Continuous batching, and why it matters

Naive static batching collects a fixed number of requests, runs them to completion together, and starts the next batch. It is simple and it wastes a great deal.

Requests finish at different times, because output lengths differ. In a static batch, a slot whose sequence has finished stays idle until the longest sequence in the batch completes, and a request arriving a moment after the batch starts waits for the entire batch.

Continuous batching — the standard in every serious serving stack — operates per generation step instead. When a sequence finishes, its slot is freed immediately and a queued request takes it at the next step. There is no batch boundary to wait for.

This is why a well-run inference service can hold high utilisation with wildly variable request shapes, and why the throughput figures published for serving frameworks are so much higher than naive batching would suggest.

Where batching stops helping

Throughput does not rise indefinitely with batch size. Three things stop it.

The first is the crossover from memory-bound to compute-bound. Once the batch is large enough that arithmetic units are saturated, adding sequences no longer amortises anything — you are simply queueing more work through a full pipeline, and per-request latency rises in proportion.

The second is memory. Every batched sequence needs its own KV cache, resident for its whole lifetime. Batch size and context length multiply against a fixed memory budget, so a service running long-context requests can batch far less than one running short ones. GPU memory for inference covers that budget.

The third is latency tolerance. A larger batch means each individual sequence advances more slowly per wall-clock second, because every step processes more work. Throughput and per-request speed pull against each other, and there is no configuration that maximises both.

What this means for the prices you see

Per-token prices are set assuming high batch utilisation. A provider quoting a low price per million tokens is quoting the amortised cost of a well-packed server, not the cost of running your request alone.

Several pricing structures follow directly. Batch or asynchronous tiers, where you accept a delayed result for a discount, exist because relaxing the latency constraint lets the provider pack sequences more densely. That is the whole product.

Long-context pricing reflects the opposite pressure: a long-context session consumes cache memory that would otherwise hold several short sessions, so it displaces batch capacity and is priced accordingly.

And it explains why very cheap models are cheap. Fewer active parameters means less memory traffic per token, which means more sequences fit in the same memory and bandwidth budget. DeepSeek V4 Flash at 13B active parameters is not cheap because someone chose to discount it; it is cheap because it batches densely. Per-token versus flat-rate pricing covers how that reaches your bill.

Why your latency moves without you changing anything

Batching is shared, so your request's speed depends on the other requests in flight. This is the source of most unexplained latency variance against a hosted API.

At low load, batches are small, each sequence advances quickly, and generation feels fast. At high load, batches are full, queueing appears, and the same request takes noticeably longer. Nothing about the model changed.

Time to first token is usually hit harder than the per-token rate, because a new request must wait for a scheduling slot and then complete prefill against a busy machine. Time to first token explained covers what else drives it.

The practical consequence is to measure latency across the day rather than once, and to build against a percentile rather than an average. A p50 measured at a quiet hour is not the number your users will experience. Tokens per second explained covers measuring the generation rate properly.

What you can control from the client side

You cannot set the provider's batch size, but you can affect how well your work packs.

Send independent requests concurrently rather than sequentially. Sequential calls waste the provider's batching entirely and waste your own wall-clock time; a modest concurrency limit over a queue is almost always a large improvement. Batching requests to save money covers the pattern and the discounted tiers.

Use asynchronous or batch tiers for anything not user-facing — bulk classification, backfills, documentation passes, scheduled analysis. The discount is real and the latency cost is irrelevant for work nobody is watching.

Keep output lengths bounded, since decode is the sequential part and a runaway response occupies a batch slot for its entire duration. And if you self-host, tune batch size against your actual latency requirement rather than maximising throughput, because the throughput-optimal setting will make interactive use feel sluggish.