Beam Search vs Sampling: Why Chat Models Do Not Search

Beam search dominated text generation for years and then almost entirely vanished from chat and coding models. The reason is a genuinely counterintuitive finding: searching harder for the most probable continuation produces text that people rate as worse. Understanding why explains what the sampling knobs on your API are actually for.

Greedy decoding and its blind spot

The simplest possible strategy is to take the highest-probability token at every step. This is greedy decoding, and it is what most APIs give you at temperature zero.

Its weakness is that a locally optimal choice can be globally poor. Picking the best word now may lead into a region where every continuation is mediocre, when a slightly worse word would have opened onto something far better. The model has no ability to reconsider — each token is committed the moment it is chosen.

The sequence greedy decoding produces is therefore not the most probable sequence. It is merely the sequence of most probable individual steps, which is a different and weaker thing.

What beam search does about it

Beam search keeps several partial sequences alive at once. With a beam width of five, it maintains five candidate continuations, extends each by every plausible next token, scores the resulting sequences by total probability, and keeps the best five.

At the end it returns the highest-scoring complete sequence. It cannot examine every possible sequence — that space is astronomically large — but it explores enough to routinely find continuations that greedy decoding walks straight past.

For tasks with one broadly correct answer, this works well. Machine translation, speech transcription and short summarisation all improved measurably with beam search, and it was the obvious default for a decade.

Why higher probability meant worse text

Then people applied it to open-ended generation and found something odd. Beam search reliably found higher-probability text, and human raters reliably preferred the sampled alternative.

The reason is that the highest-probability text is the blandest text. Averaged over everything a model has read, the safest continuation of almost any sentence is a hedge, a restatement or a cliché. Genuinely informative writing takes choices that are individually less predictable.

Beam search also degenerates. Because sequences that repeat themselves score well — a phrase the model has already produced becomes highly probable to produce again — wide beams have a strong tendency to fall into loops. Search makes this worse, not better, because it is optimising for exactly the quantity that repetition maximises.

Why sampling won

Sampling draws from the distribution instead of maximising over it. If a token has 30% probability, it is chosen roughly 30% of the time.

This produces text with the statistical texture of real writing rather than the flattened average of it. It also avoids the repetition trap, since nothing is optimising for total sequence probability.

Raw sampling from the full distribution is too loose, because the tail contains many absurd tokens that collectively hold non-trivial probability. So it is paired with truncation — cut the tail, then sample from what remains. Top-k sampling explained covers how the cut is made, and temperature, top-p and sampling covers how the knobs combine.

Where search-like decoding survives

Beam search itself is largely gone from general-purpose APIs, but the underlying idea reappears in three places.

Constrained decoding does a limited kind of lookahead. When a grammar governs the output, the decoder must avoid tokens that would make a valid completion impossible, which requires reasoning about the future rather than only the present. Structured outputs and JSON mode covers how that is implemented.

Best-of-n generates several complete responses independently and picks one with a verifier or a scoring model. This is search at the response level rather than the token level, and it works because the selection criterion is external quality rather than the model's own probability.

Reasoning models internalise the idea. Rather than searching over token sequences, they are trained to explore alternatives in their generated reasoning and revise, which spends compute on getting the answer right in a way beam search never could. Reasoning models explained covers the trade.

What this means for your settings

Practically, you have temperature and truncation, and beam search is not an option worth hunting for. The decisions in front of you are simpler than the theory suggests.

For code, structured output and extraction, use temperature at or near zero. There is usually one right answer, variety is pure risk, and greedy decoding is what you want. This is also the closest you get to reproducibility, though not fully — determinism and seeds covers why identical requests still vary.

For prose, brainstorming and anything where you will pick among options, raise temperature moderately and let sampling do its job. If you need the best of several, generate several and choose, rather than trying to make one generation better by tuning.

The takeaway

The useful principle is that maximising sequence probability is not the same as maximising quality, and the two diverge exactly where the task is open-ended. Search helps when there is a single correct target and hurts when there is not.

So spend your effort in the right place. If you find yourself tuning decoding parameters to fix a quality problem, the fix is almost always elsewhere — in the prompt, in what context you supplied, or in verifying the output afterwards. Sampling settings decide texture; they do not decide correctness.