Latency-Adjusted Model Scoring: When Fast Beats Smart

A benchmark harness does not care whether an answer arrived in two seconds or two minutes. A developer waiting on an autocomplete cares about almost nothing else.

That gap is why capability rankings and real-world satisfaction diverge so often. Latency is not a secondary consideration for interactive workloads; past a threshold it is the primary one, and no amount of extra capability compensates.

Three different numbers people call latency

Confusing them causes most of the bad decisions in this area.

Time to first token is how long before anything appears. For streaming interfaces this is the number the user experiences as responsiveness, and it is mostly determined by queueing, prompt processing and the length of your input.

Tokens per second is the generation rate once output starts. It determines how long a long answer takes to finish and matters more as output length grows.

Total time to completion is what an agent or a batch job actually experiences, because nothing downstream can start until the whole response exists. For non-streaming consumers this is the only number that matters, and it is roughly time to first token plus output length divided by rate.

A model can be excellent on one and poor on another. Reasoning models in particular often have acceptable generation rates and very long total times, because they emit a large volume of intermediate tokens before the answer. Inference latency explained covers the mechanics.

Input length is part of the latency

Prompt processing is not free, and it scales with how much you send. A model with a 1M context window will not process a million tokens instantly, and the time to first token on a very long prompt can dwarf the generation time.

This matters because the models with the largest windows are frequently chosen for exactly the workloads that fill them. Kimi K3, GLM-5.2, both DeepSeek V4 variants and MiniMax M3 all ship 1M context; using it fully has a latency price that no leaderboard reports.

Prompt caching changes the calculation substantially. A cached prefix skips reprocessing, which cuts time to first token as well as cost. If your requests share a long stable prefix, measure with the cache warm, because that is your real steady state. Prompt caching explained covers how the prefix matching works.

Building the adjusted score

The useful construction is not a formula that blends quality and speed into one number. It is a constraint plus a ranking.

Set a latency budget from the interaction, not from the model. Inline completion tolerates a few hundred milliseconds. A chat response tolerates a second or two to first token. A code review comment on a pull request tolerates a minute. An overnight batch tolerates hours.

Then exclude every model that misses the budget at your input length and concurrency, and rank the survivors on quality or cost per success. This is more honest than a weighted blend because latency requirements are genuinely threshold-shaped: below the budget the difference barely matters, above it the model is unusable.

Where you do want a single number, cost per successful outcome per unit time is workable for batch pipelines with a throughput target. For interactive work, resist it. Cost-adjusted scoring covers the quality-per-currency half.

Measure under your own concurrency

Published latency figures and single-request tests are measured on an idle path. Your production numbers will be worse, and how much worse depends on load.

Serving systems batch requests to use hardware efficiently, which means your latency depends on how many other requests are in flight. A model that looks fast in a solo test can degrade sharply under concurrency, and one that looks mediocre may hold up better.

Measure percentiles, not means. The mean is dominated by the common case and the user experience is dominated by the tail. Track p50, p95 and p99, and set the budget against p95.

Also measure across the day. Provider capacity is shared, and time to first token at peak hours is a different number from the same measurement at three in the morning. Rate limits and retries covers what happens when that capacity runs out.

Where latency dominates the decision

Inline completion is the clearest case. If a suggestion arrives after the developer has typed the next line, it is worthless no matter how good it was. Here the smallest model that clears a quality floor wins outright.

Agent loops are the subtler case. An agent taking twenty steps multiplies per-call latency by twenty, and the compounding turns a two-second difference per call into a forty-second difference per task. Agent loop anatomy shows where the calls accumulate.

Interactive debugging sits in between. A developer will wait longer for an answer they believe is better, but only up to about a minute, after which they switch context and the session is lost anyway. Choosing for low latency covers the shortlist.

Where it does not

Batch and scheduled work is the obvious exception. A nightly documentation pass, a bulk classification job or an offline evaluation run has hours of slack, and optimising latency there trades away quality or cost for nothing.

Anything with a human review step before the output is used also has slack, because the human is the bottleneck. A pull request review that takes ninety seconds to generate is not slower than one taking thirty if the reviewer looks at it an hour later.

The decision rule is simple enough to state in one line: if a human or a downstream step is waiting on the response, latency is a hard constraint and quality is the tie-breaker. If nothing is waiting, invert it.