Self-Consistency Decoding: Ask Five Times, Take the Majority

Ask a model the same question five times with sampling enabled and you get five reasoning paths. Some reach the same conclusion, some do not. Self-consistency takes the answer that appears most often and discards the rest.

It is one of the simplest reliability techniques available, it requires no training and no special model support, and it is easy to apply where it does nothing useful. The distinction is worth understanding before you multiply your bill by five.

Why voting works when it works

Sampling introduces randomness at each token. A model that is confident produces nearly the same output every time, because the probability mass is concentrated and the sampled choices rarely diverge.

A model that is uncertain produces different outputs, because several continuations are plausible at the branch points. Those divergent paths lead to different conclusions.

The useful asymmetry is that wrong answers tend to scatter while right answers tend to converge. There is usually one correct answer and many distinct ways to be wrong, so the correct one accumulates votes and each error gets one. Majority vote exploits that asymmetry directly. Temperature, top-p and sampling covers the knobs that control divergence.

The requirement people skip

Voting needs comparable answers. If the task produces a single value — a number, a label, a yes or no, a chosen file path — comparison is trivial and the technique applies cleanly.

If the task produces free-form prose, there is nothing to count. Five explanations of a design decision will be five different strings even when they agree completely, so a naive majority vote finds no majority at all.

You can sometimes recover this by extracting a structured field from each response and voting on that — the verdict, not the justification. But if the value of the output is the prose itself, self-consistency is the wrong tool. Structured outputs and JSON mode covers making the extractable field reliable.

Temperature has to be non-zero

This trips people up in an obvious-in-hindsight way. At temperature zero the model samples greedily and every call returns the same output, so you pay five times for five identical answers and vote unanimously for whatever the first one said.

Self-consistency needs enough randomness for genuinely different reasoning paths, but not so much that the paths become incoherent. Moderate values work; very high values produce diversity that is noise rather than exploration.

Note also that even at temperature zero you may not get bit-identical results across calls, because batching and hardware scheduling introduce non-determinism at the provider. LLM determinism and seeds covers why a seed parameter is weaker than it sounds.

What it costs

Five samples cost roughly five times one sample. Input can sometimes be shared through prompt caching, which softens the input side considerably, but output tokens are paid in full for every sample and output is the expensive side of the price sheet.

The interaction with reasoning models is where budgets get destroyed. A reasoning model emits a long internal trace before answering, and self-consistency multiplies that trace by the sample count. Five samples on a hard problem can be a startling number of billed output tokens for one answer. Test-time compute explained covers the general trade.

Latency is more forgiving. Samples are independent, so issuing them concurrently keeps wall-clock time close to a single call — provided your rate limits allow the burst. Rate limits and retries covers what happens when they do not.

Where it genuinely pays

Multi-step problems with a single verifiable answer. Arithmetic and unit conversion. Logical deduction. Classification where the label set is fixed and the boundaries are genuinely ambiguous.

Extraction tasks are a good fit that people overlook. Pulling a specific value out of a messy document is exactly the shape where a model is often right and occasionally distracted, and voting suppresses the occasional distraction cheaply.

It also gives you a confidence signal for free. A unanimous vote and a three-to-two split mean different things, and routing the split cases to a human or a stronger model is a better use of the information than simply taking the winner. Human-in-the-loop design covers building that escalation path.

Where it fails

Systematic errors do not vote away. If the model consistently misreads an ambiguous instruction, all five samples share the misreading and the majority is confidently wrong. Voting reduces variance, not bias.

Knowledge gaps are the same story. A model that does not know a fact will invent it five times, sometimes even converging on the same invention, which produces a unanimous vote for a fabrication. Agreement is not evidence of correctness. Why LLMs hallucinate covers why the fabrication is fluent.

And where a real verifier exists, use the verifier instead. Running the tests beats counting opinions about whether the code works. Voting is for the case where nothing external can adjudicate.

A practical configuration

Start with three samples rather than five. Most of the benefit arrives early, and three is the smallest count that can break a tie.

Apply it selectively rather than globally. Use a single call by default, and escalate to voting only on requests you have identified as high-stakes or historically unreliable — the ones where a wrong answer is expensive rather than merely annoying.

Then measure whether it moved anything. Run your evaluation set with one sample and with three, and compare accuracy against the token bill. If the gain is a fraction of a percent, you have bought very little for a threefold cost, and that outcome is more common than the technique's reputation suggests.