Time to First Token: What It Measures and What Moves It

Time to first token is the interval between your request leaving the client and the first piece of generated content arriving back. In a streaming interface it is the delay the user experiences as the thing sitting there doing nothing, and it is the number that decides whether an application feels responsive.

It is also routinely misattributed. Teams see a slow TTFT, conclude the model is slow, and switch to a smaller one — which sometimes helps and often does not, because the dominant term is usually prompt length, queueing or geography rather than model size. The useful thing is knowing which of those you are actually paying.

What the number includes

Be precise about the boundaries, because the same label gets applied to four different measurements and they differ substantially.

The full path is: DNS and TCP and TLS setup, the request travelling to the server, the server queueing it, prefill computing over your prompt, the first token being sampled, and that token travelling back. Most client libraries report the interval from the moment the request is issued to the moment the first content-bearing chunk is read, which folds all of that together.

Two traps live in that measurement. The first byte of the response is not always the first token — some servers send headers, a role delta or a keep-alive comment before any content, so timing to first byte can flatter the result by a meaningful margin. And a reasoning model may emit a long internal segment before anything user-visible, which means TTFT and time-to-first-useful-output are different quantities. Decide which you care about and measure that one consistently. The wire-level detail is in how server-sent events actually work.

Prefill is the dominant term

Before a model can produce a token it must process every token of your prompt. That pass, the prefill, builds the key-value cache the generation phase reads from, and it is where TTFT mostly goes.

Prefill is compute-bound and highly parallel, which distinguishes it from generation. All prompt tokens are processed together rather than one at a time, so a GPU can saturate its arithmetic units. This is why prefill throughput is measured in thousands of tokens per second while generation is measured in tens.

The consequence is a straightforward scaling relationship: doubling the prompt roughly doubles prefill time, at least until attention costs start to matter. Attention is quadratic in sequence length, so at long contexts the growth becomes steeper than linear. A prompt in the hundreds of thousands of tokens has a TTFT dominated by attention over that history, which is a large part of why very long contexts feel sluggish even when the generation itself is quick.

The practical reading: if your TTFT is poor, look at your prompt size before you look at your model. A system prompt that grew to twenty thousand tokens of examples is paid on every single request, and trimming it is usually the cheapest available win. The mechanism behind the cache being built is covered in the KV cache explained.

Queueing and batching

Your request does not go straight to a GPU. It joins a queue, and on a busy server the wait can exceed the compute time by a wide margin.

Modern servers use continuous batching, where new requests are slotted into a running batch as slots free up rather than waiting for a whole batch to finish. That is a large improvement over static batching, but it does not eliminate the wait: an arriving request still waits for the next scheduling step, and if every slot is occupied by long-running generations it waits for one to complete.

Chunked prefill changes the picture again. Servers split a long prefill into pieces so that other requests can be interleaved, which keeps one enormous prompt from blocking everyone else. The trade-off is that your own prefill takes slightly longer while being kinder to concurrent requests, and it is a scheduler setting rather than something the caller controls.

What this means for you is that TTFT is a function of the provider's current load, not only of your request. The same prompt to the same endpoint can differ several-fold between a quiet period and a busy one, which is why a single measurement tells you almost nothing. Continuous batching covers the scheduling side in detail.

Prompt caching changes the arithmetic

If a prefix of your prompt has been processed before and the server still holds its key-value cache, prefill can skip it entirely. The saving is proportional to how much of the prompt is a cache hit, and for agent loops with a large fixed system prompt that fraction can be most of it.

This is the largest single lever available to a caller, and it is structural rather than a setting. Caches match on prefixes, so anything that varies must go after everything that does not. A timestamp at the top of the system prompt invalidates the cache on every request and costs you the entire saving.

Cache lifetimes are short and provider-specific, so a low-traffic endpoint may never hit. Reuse also depends on landing on a server that holds the entry, which you do not control. Treat caching as a large average improvement with high variance rather than a guarantee, and measure hit rate separately from latency. The details are in prompt caching explained.

Network distance sets the floor

Nothing you do at the model layer beats physics. A round trip between continents costs a fixed amount of time, and TLS negotiation costs more round trips on top of it if the connection is new.

For short prompts this can be the majority of TTFT. A request whose prefill takes a few tens of milliseconds and whose network path costs a couple of hundred is a network problem wearing a model problem's clothes, and switching models will not touch it.

The fixes are ordinary engineering. Keep connections alive so TLS is not renegotiated per request. Reuse a connection pool rather than creating a client per call. Put your inference calls in a region near the provider's endpoint rather than near your users, since the model call is usually the long pole. And if you proxy, count the extra hop honestly. Keeping a warm pool of connections to the endpoint is boring work that pays back on every short request.

Why TTFT and throughput pull against each other

A server operator has one fundamental dial: batch size. Larger batches use the hardware more efficiently and produce more total tokens per second across all users. They also make any individual request wait longer to be scheduled and share memory bandwidth with more neighbours.

So tuning for aggregate throughput degrades individual TTFT, and tuning for TTFT wastes capacity. Providers pick a point on that curve, and different endpoints from the same provider — an interactive tier against a batch tier — are usually the same model at different points on it.

This is why a cheaper endpoint is often a slower one in a specific way: it is running larger batches. It is also why your own latency degrades when someone else's traffic spikes, with no change on your side. If your workload is genuinely offline, accepting a worse TTFT for a lower price is a good trade — the shape of that decision is in throughput versus latency trade-offs.

Measuring it properly

Most TTFT numbers quoted in comparisons are unusable because of how they were collected. A defensible measurement takes some care.

Measure with your real prompts, at your real lengths. TTFT scales with prompt size, so a benchmark using a twenty-token prompt tells you nothing about an application that sends eight thousand. Use a representative sample of production requests rather than a synthetic one.

Report percentiles, not a mean. The distribution is skewed by queueing, so the median tells you about a quiet moment and the p95 tells you what your users complain about. A mean sits between them and describes nobody's experience.

Sample across time and load. Run continuously for at least a day, because provider load follows a daily cycle and a Tuesday morning number does not predict Thursday evening. Separate cold and warm cache runs and report them independently rather than blending them, since mixing the two produces a bimodal distribution whose average is meaningless.

Instrument the segments you can see. Log DNS and connect time, time to response headers, and time to first content chunk separately. The gaps between those three tell you whether you are looking at a network problem, a queueing problem or a prefill problem, and that is the whole diagnostic. The generation phase needs its own metric alongside this one, since a fast first token followed by a slow stream is a different problem.

A decision rule

When TTFT is too high, check in this order. Is the prompt large, and is any of it removable or cacheable? Is the cache hit rate low because something variable sits near the front? Is the network path long or is a new connection being made per request? Is the p95 much worse than the median, which points at queueing rather than at anything you control?

Only after those four should you consider a smaller or faster model, because that change costs you capability and is the least likely to be the actual cause. Fix the prompt, the cache and the connection first, then measure again with the same method. If it is still slow, you have a real model or provider decision to make, and now you have the evidence to make it — choosing a model for low-latency work picks up from there.