Token Costs by Language: Why the Same Text Bills Differently

Providers price per token, not per word or per character. Since tokenizers are trained on corpora dominated by English and by Latin script, the number of tokens needed to express the same meaning varies by language — and so does the price for saying it.

Nobody set this multiplier deliberately. It falls out of how the tokenizer was built, and it shows up on your invoice whether or not you knew about it.

Where the multiplier comes from

Byte-pair encoding learns merge rules by frequency. Sequences that appear often in the training corpus get compressed into single tokens; sequences that appear rarely stay fragmented into small pieces.

Two effects then stack. The first is representation: languages that are abundant in the corpus earn many dedicated merges, so their words compress into few tokens. Languages that are scarce earn few, so their words break into fragments.

The second is encoding. UTF-8 uses one byte for ASCII, two for most European accented letters and Greek and Cyrillic, and three for most CJK characters. Byte-level tokenizers work over those bytes, so a script that needs more bytes per character starts from a worse position before frequency is even considered. Unicode and tokenization covers those layers in detail.

The combination means English tends to be the cheapest thing you can say, other Latin-script European languages are modestly more expensive, and scripts that are both multi-byte and underrepresented pay the most per unit of meaning.

Why quoting a table of multipliers is a mistake

You will find charts claiming a specific ratio for each language. Treat them as folklore.

The multiplier depends on the tokenizer, and every model family trains its own. A model built with heavy multilingual data during tokenizer training will show a much smaller penalty than one built primarily on English text, even for the same language and the same input. Tokenizer differences across models covers how wide that spread gets between families.

It also depends on your text. Formal prose, chat messages, product names, transliterated terms and code-switched sentences all tokenise differently within the same language. A multiplier measured on news articles will not predict your support tickets.

So the useful move is not to look up a number but to measure the one that applies to your traffic, which takes an afternoon.

How to measure your own

Take a genuine sample of your production text, a few hundred items per language, drawn from real traffic rather than translated test sentences. Translation tends to produce cleaner, more formal text than users write.

Run each item through the tokenizer of the model you actually call, and record tokens alongside a language tag. Compare tokens per item, or tokens per character, against your baseline language.

The number that matters is not the ratio itself but its effect on your bill. Multiply the ratio by the share of traffic in that language, and by your per-token price, and you get the actual annual difference. Often it is smaller than the ratio makes it feel; occasionally it is the largest single line in a support-automation budget.

Repeat the measurement when you change models. A provider migration can move the multiplier substantially in either direction, and it is easy to miss because the aggregate token count moves for several reasons at once.

Code has its own profile

Source code is not a natural language and does not behave like one.

Modern tokenizers are tuned for it. Runs of leading whitespace usually merge into single tokens, so indentation is far cheaper than its character count suggests. Common keywords, operators and punctuation pairs are single tokens. Popular library and function names often are too.

What is expensive in code is what is rare: long unique identifiers, hashes, base64 blobs, minified bundles, UUIDs and generated data. These fragment badly because no merge rule ever fired for them. A file of generated fixtures can cost several times what an equivalent length of hand-written source costs.

That has a direct consequence for coding agents. Pulling a lockfile, a build artefact or a large fixture into context is disproportionately expensive relative to how much it helps. Reducing token usage and context compaction strategies both start from excluding this material rather than compressing it.

The multiplier hits output too, and asymmetrically

Output tokens are priced several times higher than input tokens across essentially every provider. A language penalty therefore costs more when it lands on generation than on reading.

This changes where to spend effort. If your application reads long documents in one language and answers briefly, the penalty is diluted by cheap input pricing. If it generates long responses in an expensive-to-tokenise language, the penalty compounds against the expensive side of the bill. Input versus output token pricing covers the ratio and why it exists.

It also affects limits rather than just cost. Output caps and context windows are counted in tokens, so a response of a given length in an expensive language consumes more of the budget, and long-form generation truncates sooner than it does in English.

What to actually do about it

If a language multiplier is material to your economics, the honest options are a short list.

Compare candidate models on your own multilingual sample rather than on English benchmarks, because tokenizer efficiency for your languages is a genuine selection criterion and it is invisible in published evaluations.

Keep prompts and system instructions in whichever language tokenises cheapest if the model handles the mix well — many do, and a system prompt is paid on every single request.

Cache aggressively. A stable system prefix that is expensive to tokenise is exactly the thing prompt caching is best at, so the penalty is paid once rather than per call. And be explicit about response length, since output is where the multiplier bites hardest. Caching strategies to cut cost covers the rest of that pattern.