Continuous Batching: How Servers Keep GPUs Busy

Take eight requests, run them as one batch, and wait for all eight to finish before starting the next eight. That is static batching, and it wastes most of the hardware most of the time.

The reason is that generation lengths differ wildly. One request stops after twenty tokens, another runs to two thousand. Under static batching the seven short ones sit in the batch doing nothing while the long one finishes, and no waiting request can start. Continuous batching removes that constraint, and it is the reason serving throughput improved sharply without any change to the models themselves.

The mechanism

Generation happens one step at a time. At each step, every active request produces exactly one token. That step boundary is the opening continuous batching exploits.

Instead of fixing the batch membership when the batch starts, the scheduler re-evaluates it at every step. A request that emitted a stop token is removed. A request waiting in the queue is admitted. The next step runs with the new membership, and the batch composition changes continuously as work arrives and completes.

Nothing about the model changes. The same weights are read, the same attention is computed. What changes is that a slot freed by a finished request is refilled immediately rather than at the end of the batch, so the hardware rarely runs a step with idle capacity.

This is sometimes called in-flight batching or iteration-level scheduling. The names describe the same idea: schedule per decode step, not per batch.

Why it helps so much

Under static batching, effective utilisation is governed by the longest request in each batch. Mix a two-thousand-token generation with seven twenty-token generations and you are paying for eight slots to produce roughly the work of one.

Continuous batching converts that dead space into completed requests. Under mixed traffic — which is all real traffic — the improvement is large, and it lands mostly as reduced queueing rather than faster individual generation.

The subtler benefit is on the tail. A short request arriving behind a long one no longer waits for it. It joins at the next step and finishes on its own timescale, which pulls in the ninety-ninth percentile far more than the median. The throughput and latency trade-off is exactly what this softens: you gain utilisation without forcing every user onto the slowest request's clock.

Prefill is the complication

Admitting a new request is not free. Before it can join the decode loop it needs its prompt processed, and prefill is a heavy compute operation that scales with prompt length.

If the scheduler runs a full prefill in the middle of a decode sequence, every request already generating stalls for the duration. Users see a stutter that has nothing to do with their own request — someone else arrived with a long prompt.

Servers handle this in two ways. Chunked prefill splits a long prompt into pieces and interleaves them with decode steps, spreading the cost so no single step is enormous. Prioritisation policies decide whether an arriving request should wait rather than disrupt requests already in flight.

Both are trade-offs between the newcomer's time to first token and everyone else's smoothness. There is no setting that avoids the choice, which is why time to first token and inter-token latency often move in opposite directions when you tune a server.

Memory is what limits admission

The scheduler cannot admit requests indefinitely. Each active request holds a key-value cache that grows with every token it generates, and that memory is occupied for the whole life of the request.

So admission is really a memory allocation decision. The server admits a new request only if it believes there is room for its cache to grow. Guess too conservatively and capacity sits unused; guess too aggressively and the system runs out of memory mid-generation.

When it does run out, the server must preempt — evict a request's cache and either recompute it later or swap it out. Preemption is expensive and visible, because the evicted request either pauses or repeats work it already did. Sustained preemption is a sign the deployment is over-admitting for its real context lengths.

This is why paged attention and continuous batching arrived together. Fine-grained cache allocation is what makes aggressive admission safe, and the cache itself is the resource being managed.

What it looks like from outside

If you call a hosted API you never configure any of this, but you observe it constantly.

Variable time to first token across identical requests is queueing and admission. Generation that starts smooth and then stutters is usually other requests joining the batch, not anything about your prompt. A provider that stays responsive as load climbs and then degrades abruptly is hitting a memory ceiling rather than a compute one.

None of it is under your control, but it does change how you interpret a slow request. Retrying immediately after a timeout adds a fresh request to a queue that is already saturated, which is why backoff policy matters more than retry count.

If you run the server yourself

The settings that matter are the maximum number of concurrent sequences, the memory fraction reserved for the cache, and whether chunked prefill is enabled. Defaults are tuned for short prompts and will over-admit if your workload is long-context.

Tune with your own traffic shape. A benchmark that sends uniform prompts of uniform length hides the exact behaviour continuous batching exists to handle, and will suggest a concurrency limit that collapses under mixed traffic.

Watch preemption counts and queue wait time as first-class metrics, not just tokens per second. Aggregate throughput can look excellent while a minority of requests are being evicted and recomputed repeatedly.

The takeaway

Continuous batching is why a single deployment can hold high concurrency without the queue collapsing, and it is now the default in every serious serving stack. Assume it is on.

What it does not do is make any individual request faster. It removes waiting, not work. If your problem is that generation itself is too slow, the lever is elsewhere — a smaller model, shorter output, or speculative decoding — not the scheduler.