Logits and Softmax: What a Model Really Outputs

A language model does not produce text. It produces one number for every token in its vocabulary — perhaps 150,000 of them — and everything else is a decision made after the model has finished. Understanding that boundary clears up most confusion about temperature, sampling, logprobs and why a confident-sounding answer tells you nothing about whether it is true.

What the final layer emits

The last thing the network does is project its internal representation onto the vocabulary. The result is a vector of raw scores called logits, one per token, and they are unbounded — positive, negative, any magnitude.

A logit is not a probability and is not interpretable on its own. It only means something by comparison: a logit of 8 next to a best of 8.1 is nearly a tie, while a logit of 8 next to a best of 20 is not in contention.

Every token in the vocabulary receives one, including thousands that are obviously absurd in context. There is no filtering step inside the model; it scores everything, every time. Tokenisation explained covers what those vocabulary entries actually are, and it is worth knowing that many are fragments rather than words.

What softmax does

Softmax converts that vector of scores into a probability distribution. It exponentiates each score and divides by the total, which guarantees every value is positive and the whole set sums to one.

The exponentiation is the part with consequences. It magnifies differences: a gap of two in logit space becomes a much larger ratio in probability space. A token scoring a little higher than the rest ends up dominating the distribution rather than merely leading it.

The flip side is that the long tail never quite reaches zero. Every absurd token retains some tiny probability, and with enough draws a tiny probability eventually fires. That is the entire reason truncation methods exist.

Temperature is a division before softmax

Temperature is applied to the logits before softmax, by dividing every one of them by the temperature value.

Dividing by a number below one spreads the scores apart, so after exponentiation the leader dominates even harder and the distribution sharpens. Dividing by a number above one compresses them together, flattening the distribution and giving unlikely tokens a real chance.

Temperature zero is a special case handled separately, since dividing by zero is undefined — implementations treat it as "take the highest logit". Note that this changes only which token gets chosen, never what the model believes. Temperature, top-p and sampling covers how the knobs interact.

Why probability is not confidence

This is the most consequential misunderstanding in the area. A 97% probability on a token means the model finds that token overwhelmingly likely to come next given the training distribution. It does not mean the resulting statement is 97% likely to be true.

Consider a prompt asking for the signature of a plausible-sounding but nonexistent library function. The model has seen thousands of similar signatures, the pattern is unambiguous, and it will assign very high probability to a fluent, well-formed, entirely fabricated answer.

High probability means "this is the expected continuation". Fabrication is the expected continuation whenever the pattern is strong and the fact is absent. Why LLMs hallucinate works through the mechanism.

What logprobs are actually good for

Many APIs will return the log-probabilities of chosen tokens and the top alternatives. They are genuinely useful, provided you use them for the right thing.

They work well for classification-style tasks with a constrained answer space. If you ask for one of three labels, comparing the probability assigned to each is far more informative than the label alone, and it gives you a threshold to route uncertain cases to a human. Human-in-the-loop design covers where to put that boundary.

They work poorly as a truthfulness score on free text. A long answer has a probability for every token, most of them structural, and averaging them tells you mostly how fluent the sentence was. A confidently wrong answer usually has a high average.

How this constrains structured output

Constrained decoding is implemented at exactly this layer. To guarantee valid JSON, a system tracks what the grammar permits next and sets the logits of every disallowed token to negative infinity before softmax runs.

Those tokens then receive zero probability and cannot be sampled, regardless of sampling settings. This is why grammar-constrained output is a hard guarantee rather than a strong suggestion, unlike asking politely in the prompt.

It is also why constrained output can degrade quality. If the model wanted to write prose and the grammar forces a brace, you get a syntactically perfect object whose contents were the model's second or fifth choice. Structured outputs and JSON mode covers when that trade is worth making.

The takeaway

Keep the boundary clear: the model produces a distribution, and everything after that is a decision your stack makes. Temperature, top-k, top-p, penalties and grammars all operate on logits after the model has finished thinking.

Use logprobs for constrained choices and thresholds, not as a lie detector. And when you need to know whether an answer is correct, the answer has to be checked against something external — a test suite, a lookup, a second model with different failure modes. The distribution cannot tell you. Top-k sampling explained covers what happens to the tail next.