Best Model for Low Latency: Time to First Token Wins

When a model powers something a person is waiting on — an editor completion, a chat reply, an interactive tool — capability stops being the main variable. A brilliant answer that arrives in nine seconds loses to a good answer in one.

The mistake is optimising the wrong number. Most latency discussion focuses on tokens per second, and for interactive work that is not the number that matters.

Two different measurements

Time to first token is how long from request to the first piece of output. This is what a user experiences as responsiveness. Until the first token arrives, the interface is dead.

Throughput is tokens per second once generation is underway. This determines how long a long answer takes to finish.

For interactive work, time to first token dominates perception. Once text starts appearing, people read along and tolerate moderate speed. A model that starts in 200 milliseconds and generates at forty tokens a second feels dramatically faster than one that starts in three seconds and generates at a hundred, even though the second finishes a long answer sooner.

LLM inference latency explained covers where the time actually goes.

What drives time to first token

Prompt length. The model must process your entire input before emitting anything. A 100K-token prompt has a floor no model beats. This is usually the biggest lever and the one teams control most directly — trimming a bloated system prompt often does more than switching models.

Active parameters. Fewer parameters per token means less computation. This is why small dense models feel snappy and why the cheap tier of a model family is often the right pick for interactive work regardless of capability.

Prompt caching. If the provider caches your prefix, repeated calls with the same system prompt skip most of the prefill. This can be transformative for chat, where the prefix is stable and only the last turn changes. Prompt caching explained covers making your prompts cacheable, which mostly means keeping the variable parts at the end.

Queueing. Under load, waiting for a slot can exceed the model's own processing time. This is a provider capacity question rather than a model question, and it is why the same model can feel fast at 3am and slow at 3pm.

Model choices

• Smallest model that clears your quality bar. For completion, classification and short transformations this is often much smaller than teams assume. DeepSeek V4 Flash at roughly 13B active is fast and cheap.
• Dense over mixture-of-experts, at similar capability. Dense models have more predictable latency because every token costs the same. MoE routing adds variance, and variance is what users notice — a p99 that is five times the median feels broken even if the median is fine.
• Avoid reasoning-heavy modes. Extended thinking dramatically increases time to first useful output. If the model has effort levels, interactive paths should use the lowest that works. Reasoning models explained covers the trade.

Things that help more than model choice

Stream everything. Streaming does not reduce total time but transforms perceived time, because the interface comes alive at first token instead of at completion. If you are not streaming an interactive response, fix that before comparing models. Streaming and server-sent events covers implementation.

Shorten the prompt. System prompts accumulate instructions nobody has audited in months. Cutting a 4,000-token preamble to 800 reduces prefill on every single request.

Cache aggressively. Put stable content first and variable content last so the cacheable prefix is as long as possible.

Speculate. For predictable interactions, start the request before the user finishes — on a debounce during typing, for instance. Discarding occasional wasted calls is often cheaper than the latency you save.

Measure the tail, not the average

Average latency hides the problem. Users remember the slow requests, not the median ones.

Track p50, p95 and p99 for time to first token separately from total time. A model with a good median and a terrible p99 produces an interface people describe as unreliable, and no amount of average-case tuning fixes that impression. If your p99 is several times your p50, investigate queueing and routing variance before changing models.

Where latency and cost pull in the same direction

Latency work and cost work usually conflict — faster generally means bigger and more expensive. Interactive workloads are the exception, and it is worth exploiting.

Everything that reduces time to first token also reduces spend. A shorter prompt costs fewer input tokens and prefills faster. Prompt caching cuts both the bill and the wait. A smaller model is cheaper per token and starts sooner. Lower reasoning effort emits fewer tokens and returns quicker.

That alignment means an interactive path is the highest-return place to optimise in most systems. The same afternoon of work improves the product and the invoice, which is rare enough to prioritise. Reducing token usage covers the prompt-side work in detail.

Do not let batch work share the interactive path

A common architectural mistake is routing everything through one configuration. Background jobs then compete with interactive requests for the same capacity, and users experience queueing caused by work nobody is waiting on.

Separate them. Interactive requests get the small fast model, aggressive caching, low reasoning effort and priority capacity. Batch work gets whatever is cheapest and can wait. The two have opposite requirements and should not share a configuration or a queue.

This is also where provider choice starts mattering more than model choice — the same model behind a saturated endpoint is slow regardless of its specifications.