Vocabulary Size Trade-offs in Language Model Tokenizers

Vocabulary size is one of the few architectural numbers a model card states plainly, and one of the least understood. It is not a quality rating. It is a dial that trades sequence length against parameter count and rare-token quality, and every position on it costs something.

What the number counts

The vocabulary is the fixed set of tokens the model can read and emit. Byte-pair encoding builds it by merging frequent adjacent pairs until the target size is reached, so the size directly determines how many merge rules exist. Byte-pair encoding explained covers that training procedure.

More entries means more merges, which means longer sequences get compressed into single tokens. Fewer entries means text fragments into smaller pieces.

Two parts of the model are sized by this number. The embedding table maps each vocabulary entry to a vector, and the output layer produces a score for every entry at every generation step. Both grow linearly with vocabulary size, and in smaller models they can be a substantial share of total parameters.

The case for a larger vocabulary

The direct benefit is shorter sequences for the same text. That is worth more than it first appears because it compounds through several costs at once.

Attention cost grows with the square of sequence length, so a ten percent reduction in tokens is more than a ten percent reduction in prefill work. The KV cache also shrinks proportionally, which means more concurrent sessions fit in the same memory. KV cache explained covers why that occupancy is usually the binding constraint on serving.

Shorter sequences also mean fewer generation steps for the same output, and since decode is sequential, that is a direct latency win. A model emitting a paragraph in fewer tokens finishes sooner at the same tokens-per-second rate.

And since you are billed per token, a more efficient vocabulary is cheaper for identical work. This is a real and frequently overlooked axis when comparing providers whose headline per-token prices look similar.

The case against

The costs are less visible but real.

Parameters spent on embeddings and the output layer are parameters not spent on the layers that do reasoning. For a large model this is a rounding error; for a small model targeted at a single device it can be a meaningful fraction of the budget, which is why compact models often ship smaller vocabularies than their larger siblings.

The output layer also has to produce a score across every entry at every step, so a very large vocabulary adds work to the part of inference that is already the sequential bottleneck.

The subtler cost is training signal. A fixed corpus divided across more vocabulary entries means each entry appears fewer times. Rare tokens end up with weakly trained representations, and prompts that land on them can produce erratic behaviour. Pushing the vocabulary larger creates more of these thinly trained entries, not fewer.

Multilingual pressure pushes the number up

The trade-off changes shape once a model has to serve many languages.

A vocabulary sized for English spends nearly all its merges on English and Latin-script patterns. Text in other scripts then fragments, costing more tokens for the same meaning and consuming more of the context window. Token costs by language covers how large that gap gets.

Growing the vocabulary is the standard fix, because it creates room for merges in scripts that would otherwise be starved. This is why multilingual models tend towards larger vocabularies than English-centric ones of similar size.

But the fix is a redistribution, not a free win. Merges allocated to one language are merges not allocated to another, and the allocation follows the composition of the tokenizer training corpus. Two models with identical vocabulary sizes can have very different efficiency profiles depending on what that corpus contained.

Code changes the calculation again

Source code rewards vocabulary differently from prose. Runs of leading whitespace, common keywords, operator pairs and popular identifiers all compress well if the tokenizer training corpus contained enough code.

Indentation is the clearest case. A tokenizer with dedicated entries for whitespace runs bills four spaces as one token; one without bills four. Across a large repository read by a coding agent, that difference alone is significant.

The reverse holds for generated content. Hashes, UUIDs, base64 and minified bundles fragment regardless of vocabulary size, because no plausible corpus makes those sequences frequent. Vocabulary tuning cannot rescue them, which is a reason to keep them out of context entirely rather than hoping the tokenizer copes.

Why you cannot compare the number across models

Vocabulary size is not portable as a quality or efficiency signal, because efficiency depends on what the merges were spent on rather than how many there are.

A model with a smaller vocabulary trained heavily on code can tokenise your repository more efficiently than a model with a larger vocabulary trained mostly on web prose. The headline number does not tell you which situation you are in.

This is also why token counts do not transfer between providers. The same prompt genuinely produces different token counts on different models, so a per-token price comparison is incomplete without measuring the counts on your own text. Tokenizer differences across models covers how to run that comparison honestly.

What to do with this

Do not treat vocabulary size as a spec to optimise for. Treat it as an explanation for behaviour you observe.

When you are comparing costs, take a representative sample of your real traffic, tokenise it with each candidate model's tokenizer, and compare effective cost as price multiplied by actual token count. That number can reorder a shortlist that looked settled on headline pricing alone.

When you are choosing a small model to run yourself, be aware that a large vocabulary consumes parameters that would otherwise do work, and check quality on your own tasks rather than assuming the bigger number is better. Best model for self-hosting covers what else to weigh there.