Top-K Sampling Explained, and Why Top-P Replaced It

Top-k sampling is the first fix anyone applies to the tail problem, and it is the one most often left at a default that nobody chose deliberately. It is worth understanding what it does, because its specific weakness is exactly what nucleus sampling was invented to solve, and knowing which to reach for saves a lot of pointless parameter fiddling.

The problem: the tail is real

At every step the model assigns a probability to every token in its vocabulary. Most of those tokens are nonsense in context, but softmax gives none of them exactly zero.

Individually these probabilities are negligible. Collectively they are not. With 150,000 vocabulary entries each holding a vanishing share, the tail can still add up to a few percent of the total mass.

Sample from the raw distribution across a thousand tokens of output and that few percent will fire repeatedly. Each time, the model emits something incoherent and must then continue plausibly from an incoherent state, which it is bad at. One bad draw is enough to derail a paragraph.

Top-k: cut at a fixed count

Top-k sorts the tokens by probability, keeps the highest k, discards everything else, renormalises the survivors so they sum to one, and samples from that.

With k set to 50, only the 50 most likely tokens are ever eligible. The entire tail is gone by construction, and the improvement over unrestricted sampling is dramatic. It is cheap, easy to reason about, and universally supported.

It is also the wrong shape of tool, and the reason is worth seeing concretely.

Why a fixed count misfires

Consider a model completing def calculate_total(items): and about to start the body. In this position the distribution is extremely concentrated — perhaps three tokens hold nearly all the mass. Keeping 50 means keeping 47 tokens the model effectively ruled out, and after renormalisation they collectively hold enough probability to occasionally win.

Now consider the same model partway through the sentence "the outage was caused by". Here hundreds of continuations are genuinely reasonable, and the distribution is flat. Keeping 50 cuts off perfectly good options and makes the output narrower than the model intended.

The same k is too generous in one position and too strict in the other. There is no value that is correct in both, because the right number of candidates depends on how confident the model is, and top-k has no way to know.

Top-p adapts to the shape

Nucleus sampling, usually exposed as top-p, fixes this by cutting on cumulative probability instead of count.

Sort the tokens by probability and accumulate from the top until the running total reaches p. Keep those, discard the rest. With p at 0.9, you keep the smallest set of tokens that between them account for 90% of the mass.

In the confident position that might be two tokens. In the ambiguous one it might be four hundred. The cut adapts automatically to how certain the model is, which is precisely the behaviour top-k could not provide. This is why top-p is the default on essentially every modern API and top-k is frequently disabled entirely.

How the knobs stack

Order of operations matters and is easy to get wrong. Temperature is applied to the logits first, reshaping the distribution. Truncation is applied afterwards, to the reshaped distribution.

That means raising temperature and lowering top-p partially fight each other: temperature flattens the distribution while top-p re-truncates it. Turning both knobs at once produces effects that are hard to attribute, which is why parameter tuning by trial and error so often goes nowhere.

If both top-k and top-p are set, most implementations apply both and you get whichever cut is more restrictive. Setting k to a large value or disabling it is usually cleaner than trying to make the two agree. Temperature, top-p and sampling covers the interactions in more depth.

What to actually set

For code generation, structured extraction and anything with one correct answer, use temperature zero or very close to it, and leave truncation alone. When you are taking the top token anyway, truncation parameters have no effect. Beam search versus sampling covers why greedy decoding is the right default here.

For prose and brainstorming, hold top-p somewhere around the provider default and move temperature only. One knob at a time gives you attributable results; two gives you superstition.

If output is repetitive or bland, resist the urge to reach for penalties first. Low temperature causes both, and it is a cleaner fix. Repetition penalties explained covers why the penalties are a blunt instrument, particularly on code.

The takeaway

Treat sampling parameters as controls on texture, not on correctness. They decide how adventurous the model is allowed to be within what it already believes; they cannot make it believe something better.

The decision rule fits in a line. One right answer means temperature zero and no truncation fiddling. Many acceptable answers means default top-p and temperature as your single dial. If you are tuning a third parameter to fix a quality problem, the problem is in the prompt or the context, and no decoding setting will reach it. Determinism and seeds covers what reproducibility you can expect once settings are fixed.