Tokenizer Differences: Why the Same Text Costs Different Amounts

Price per million tokens looks like a directly comparable figure. It is not, because a token is not a fixed unit. The same document can be materially more tokens on one model than another, and the cheaper-looking price can produce the larger bill.

What a tokenizer does

Models operate on tokens, not characters. A tokenizer converts text into a sequence of integers drawn from a fixed vocabulary, typically tens or hundreds of thousands of entries.

The vocabulary is learned from a training corpus, usually with byte-pair encoding: start from characters, repeatedly merge the most frequent adjacent pair, stop at the target size. Frequent sequences become single tokens; rare ones stay fragmented.

So the tokenizer is a compression scheme fitted to a particular corpus. Text resembling that corpus compresses well. Text unlike it does not. Byte-pair encoding explained covers the algorithm.

Where the differences show up

Languages other than English. Tokenizers trained on English-heavy corpora fragment other scripts badly. The same meaning can take several times more tokens in a language the vocabulary under-represents, which is a direct multiplier on cost and a direct divisor on effective context.

Code. Vocabularies differ in how well they cover programming constructs. One may have single tokens for common keywords and indentation runs; another may split them. Across a large codebase this compounds.

Whitespace. How leading spaces and indentation are handled varies significantly, and it matters disproportionately for Python and for deeply nested structures.

Numbers. Some tokenizers split digits individually, others group them. This affects both cost and arithmetic ability — a model seeing "1", "2", "3" separately is doing different work from one seeing "123".

Vocabulary size trades against sequence length

A larger vocabulary compresses text into fewer tokens, so a document costs less and fits further into the context window.

The cost is that the embedding and output layers scale with vocabulary size, adding parameters. There is also a data argument: rare tokens appear less often in training, so each is learned less well.

Labs pick a point on that curve, and the choices differ. This is why a model can advertise a smaller context window and still hold more of your actual document than a competitor with a larger window but a worse-fitting tokenizer.

The comparison people get wrong

Comparing $0.50 per million tokens against $0.40 per million tokens and concluding the second is cheaper assumes both count the same way. They do not.

If the first model tokenizes your content 20 percent more efficiently, it is the cheaper option despite the higher headline rate. For a workload dominated by non-English text or an unusual language, the gap can be larger than any plausible price difference.

The correct comparison takes a representative sample of your actual content, tokenizes it with each candidate's tokenizer, and multiplies the real token count by the real rate. Anything else is comparing units that are not the same size.

Effects beyond cost

Effective context shrinks. A 256K window holds less of your content if your content tokenizes poorly. Two models with identical advertised windows can differ meaningfully in how much of your data actually fits.

Character-level tasks get harder. Counting letters, reversing strings, and detailed spelling work are awkward because the model sees tokens, not characters. A word that is one token is opaque at the character level. This explains a family of failures that look like the model being stupid and are actually representational.

Rare identifiers fragment. An unusual variable name may become many tokens, consuming budget and sometimes being handled less reliably than a common one.

What to do

Measure token counts on your own content before choosing on price. Most providers publish their tokenizer or expose a counting endpoint; run a representative sample through each.

If your workload is heavily non-English, treat tokenization efficiency as a first-class selection criterion rather than a detail. It can dominate the price difference entirely. Token costs by language covers the magnitude.

And when a model fails at something character-level, check whether tokenization explains it before concluding the model is weak. Asking it to spell out a word first often fixes the task, because it forces the characters into separate tokens where they can be operated on.

Tokenization drifts between model versions too

A model family usually keeps its tokenizer stable across minor revisions, but not always across major ones. When a lab changes the vocabulary, every token count you measured becomes stale, and prompts tuned against a specific context budget can start overflowing.

This is a quiet source of breakage in pipelines that chunk documents to a fixed token size. The chunker was calibrated against one tokenizer; a model upgrade changes the counts; chunks that used to fit now exceed the limit and get silently truncated at the edge.

If you chunk, measure with the tokenizer of the model you are actually sending to, and re-measure when you upgrade. Using a generic approximation such as four characters per token is fine for rough budgeting and unsafe for anything that must fit exactly.