Tokens Per Second Explained: Reading the Number Correctly

Tokens per second is the most quoted inference metric and one of the most ambiguous. The same deployment can honestly report wildly different figures depending on whether you are counting one request or the whole server, and whether prompt processing is included.

Getting the definition straight is most of the work, because once you know which number you are looking at, the disagreements between sources stop being mysterious.

Three different numbers with the same name

Per-request output rate is how fast tokens arrive for one user once generation has started. This is what determines whether a streamed response feels fluid or laboured, and it is the number that matters for interactive work.

Aggregate server throughput is total tokens produced across all concurrent requests. It is what capacity planning and unit economics depend on, and it is typically many times the per-request rate because the server is serving many sessions at once.

End-to-end rate divides total tokens by total wall-clock time including prompt processing. It conflates two very different phases and is the least useful of the three, which does not stop it from being quoted.

A vendor reporting a large figure is usually reporting aggregate throughput. A reviewer reporting a smaller one is usually reporting per-request rate. Both can be accurate, and comparing them directly is meaningless. LLM inference latency explained covers the surrounding metrics.

What actually limits generation speed

Decode is sequential — each token depends on the one before it, so there is no way to parallelise within a single sequence. And each step is dominated by reading weights and cache from memory rather than by arithmetic.

That makes memory bandwidth the primary constraint on per-request speed. Not compute, not parameter count directly, but how fast the hardware can move data.

Active parameters matter because they determine how much has to be read per token. This is why a mixture-of-experts model with an enormous total size can generate faster than a much smaller dense model: only the active experts are read. Kimi K3 activates 104B of 2.8T, DeepSeek V4 Pro 49B of 1.6T, and DeepSeek V4 Flash just 13B — and that ordering predicts their relative generation speed far better than total size does. Dense versus MoE models compared covers the mechanism.

Context length matters too, though less than people expect. The cache grows with every token, so late tokens in a long session are slower to produce than early ones, but the effect is gradual rather than dramatic.

Why published figures disagree

Beyond the definitional problem, several measurement choices move the number substantially.

Load is the largest. A quiet endpoint produces much higher per-request rates than a busy one, because batching is shared. A figure measured at an off-peak hour is not the figure your users will see. Batch size and throughput covers that interaction.

Precision changes it. A quantised deployment reads fewer bytes per parameter and generates faster, so the same model can be reported at very different speeds depending on how it was served. Quantization explained covers the trade.

Prompt length changes it if prefill is included in the average, which is why end-to-end figures are so sensitive to test design.

And tokenisation changes it in a way almost nobody accounts for. Tokens per second is only comparable across models if a token means the same amount of text, and it does not — different tokenizers produce different counts for identical output. A model generating 20% more tokens for the same paragraph needs to be 20% faster to finish at the same moment. Tokenizer differences across models covers that spread.

What speed the user actually perceives

Comfortable reading speed sits somewhere in the region of a few tokens per second, so any modern hosted model generates faster than a human reads. Past that point, raw generation rate stops being the thing users notice.

What they notice instead is the wait before anything appears, and whether the stream stalls. A response that starts in two hundred milliseconds and runs steadily feels fast; one that starts after four seconds feels slow at any generation rate. Time to first token explained covers that half.

Consistency matters more than average speed for the same reason. A stream that pauses noticeably mid-response reads as broken even if its average rate is high, which is why tail latency deserves more attention than the mean.

Where raw rate does matter is non-interactive work: batch jobs, agent loops and anything generating long output that nobody is watching arrive. There the number translates directly into wall-clock completion time.

Where it matters most: agents

An agent runs many model calls in sequence, and the generation time of each one is on the critical path of the whole task.

Reasoning tokens compound this. Extended reasoning is generated at decode speed like any other output, so a model that thinks at length before answering pays the full sequential cost of every one of those tokens. GLM-5.2 exposing two reasoning effort levels is effectively a latency dial. Reasoning models explained covers what that buys.

So the useful metric for an agent is not tokens per second but time to a completed task — which folds in generation speed, reasoning length, tool round trips, retries and how many steps the model needs. A faster model that takes more steps can finish later. Agent loop anatomy covers where the time goes.

Measuring it properly

Measure on your own traffic, not on a synthetic prompt. Record time to first token separately from the inter-token interval, then derive the per-request rate from the generation phase alone.

Use realistic prompt and output lengths, since both change the answer. Run the measurement at several times of day, and report percentiles rather than an average, because the tail is what users complain about.

And when you compare models, compare time to a finished useful answer rather than tokens per second. That single change removes the tokenizer problem, the reasoning-length problem and the definitional problem in one step. Best model for low latency covers doing that comparison honestly.