Speculative Decoding in Practice: When It Pays Off

Speculative decoding is one of the few optimisations that makes generation faster without changing what the model produces. That combination makes it sound unconditionally good, and it is not — the speedup ranges from substantial to negative depending on the workload and the load level.

This is about the operational side: what governs the gain, why it disappears under heavy traffic, and what you can observe or control from either side of an API. The mechanism itself is covered separately.

The one number that decides everything

A small draft model proposes several tokens ahead. The full model verifies them in a single forward pass and keeps the ones it would have produced itself. The acceptance rate — how many drafted tokens survive verification — determines the entire benefit.

High acceptance means one expensive pass yielded several tokens. Low acceptance means you paid for drafting and threw the work away, and the request is slower than it would have been without speculation.

There is no acceptance rate that is universal to a model. It is a property of the model, the draft, and the text being generated together, which is why a published speedup figure tells you almost nothing about your own workload.

Why acceptance varies so much by workload

Predictable text drafts well. Code is unusually predictable in this sense — closing brackets, import blocks, boilerplate, repeated call signatures, long stretches of formatting. A small draft model gets these right almost always, so acceptance stays high and the speedup is real.

Structured output is even better. JSON with a known schema is largely punctuation, keys and delimiters, and a draft model can produce those correctly with high confidence. If you generate a lot of schema-constrained output, speculation tends to pay.

Dense original prose is the weak case. Novel reasoning, unusual naming, an argument the draft model has no basis to anticipate — acceptance drops, and the overhead can outweigh the gain. Reasoning traces sit somewhere in the middle: the surface form is often formulaic even when the content is not.

The practical consequence is that the same deployment can show a large speedup on one endpoint and none on another. Measure per workload, never in aggregate.

Load level flips the calculation

This is the part most often missed. Speculative decoding trades extra compute for fewer sequential steps, and that trade is only favourable when compute is idle.

At low batch sizes, the hardware is limited by memory bandwidth. Verification of several drafted tokens costs barely more than generating one, so the extra arithmetic is free and the latency win is large.

At high batch sizes, the hardware is already compute-saturated by the batch itself. Now the speculative work competes with real requests, and rejected drafts are pure waste. Under sustained heavy load, speculation can reduce aggregate throughput while helping nobody in particular.

Serving stacks therefore tend to enable it adaptively — speculating when the batch is small and disabling it as concurrency rises. That is the correct policy, and it is another instance of the throughput and latency trade-off being resolved dynamically rather than by a fixed setting.

What you can observe from outside

If you consume a hosted API you cannot configure any of this, but the fingerprint is visible.

Watch inter-token timing on a stream. Speculative decoding produces tokens in bursts — several arrive nearly together when a draft is accepted, then a pause while the next block is drafted and verified. Even pacing suggests plain decode; bursty pacing suggests speculation.

Watch how that pacing changes with load. A provider whose stream is bursty at quiet hours and smooth under load is almost certainly toggling speculation adaptively.

None of this changes the tokens you receive. Output is identical to what the full model would have produced, which is the property that distinguishes speculation from every quality-for-speed trade. Determinism caveats still apply for the ordinary reasons, and the same run-to-run variation you would see anyway is unrelated to speculation.

What it does not help

Speculation accelerates decode. It does nothing for prefill, so time to first token is unchanged. If your complaint is that a long prompt takes seconds before anything appears, this is the wrong lever entirely — prompt length and caching are the levers.

It also does nothing for queueing. A request waiting for admission is not decoding yet. Under a saturated queue, the fix is capacity or admission policy, not faster generation.

And it does not reduce cost per token in any billing sense. Providers price on tokens produced, not on how many forward passes it took. Speculation is a latency and utilisation optimisation on the serving side, which may show up in pricing eventually but never as a line item you control.

If you run the stack yourself

The main choices are what to draft with and how far ahead to draft.

A separate small model needs to share the tokenizer and be genuinely fast, or the drafting overhead eats the gain. Self-drafting variants avoid maintaining a second model by predicting several tokens from the main model directly, which sidesteps the tokenizer-compatibility problem and the memory cost of hosting a draft.

Draft length is a tuning parameter with a clear shape. Longer drafts win more when acceptance is high and lose more when it is low, so the right value tracks your measured acceptance rate rather than a default.

Measure acceptance rate as a first-class metric, segmented by endpoint. If it is not exported, you are tuning blind. And confirm your stack disables speculation under load — a fixed-on configuration is a common cause of throughput that degrades faster than expected as traffic climbs. Continuous batching settings and speculation settings interact, and tuning either in isolation misleads.

The decision rule

Enable it if your traffic is interactive, your batches are small to moderate, and your output is code or structured data. That combination is where the technique was designed to win and where it reliably does.

Do not expect it to rescue a saturated deployment, and do not read a vendor speedup figure as applying to your workload. Acceptance rate is measurable on your own traffic in an afternoon, and it is the only number that settles the question.