Unicode and Tokenization: Where Text Handling Goes Wrong

Most token-handling bugs are Unicode bugs wearing a different hat. A string that looks like one character to a user can be several code points, several bytes and several tokens, and each of those layers counts differently.

Getting the layers straight explains why an emoji costs more than a word, why some languages cost double, and why naive truncation produces mojibake.

Four layers that people conflate

A user sees a grapheme — one visible character. A flag, an accented letter, a family emoji.

Underneath that sits one or more code points, the numeric identifiers Unicode assigns. A family emoji is several code points joined by invisible connectors. An accented letter may be one code point or two, depending on how it was produced.

Each code point encodes to one to four bytes in UTF-8. ASCII characters take one byte; most Latin-script accented letters and Greek and Cyrillic take two; most CJK characters take three; most emoji take four.

Finally the tokenizer groups those bytes into tokens. Because modern tokenizers are byte-level, the token count follows the byte count much more closely than it follows the visible character count. Byte-pair encoding explained covers how that grouping is learned.

Every off-by-something in text handling comes from measuring at one layer and reasoning at another. Your database column length is graphemes or code points. Your API bill is tokens. They do not move together.

Why emoji are expensive

An emoji is usually four bytes, and unless that exact emoji was frequent enough in the training corpus to earn its own merge rule, those bytes tokenise as several separate pieces.

Composite emoji are worse. A skin-tone variant is a base emoji plus a modifier. A profession emoji is a person, a joiner and an object. A flag is two regional indicator symbols. Each component is its own code point, each encodes to multiple bytes, and none of the joiners are common enough to have compressed well.

The result is that a single visible glyph can consume several tokens. That is fine for a handful of emoji in a chat message and expensive at scale — if you are classifying social posts or processing emoji-heavy user content, the emoji can be a real fraction of your input bill.

Normalisation changes token counts

Unicode allows the same visible text to be encoded in more than one way. An accented letter can be a single precomposed code point, or a base letter followed by a combining accent. They render identically and compare unequal.

Since they are different byte sequences, they are different token sequences. The precomposed form is usually more common in training data and therefore tokenises more efficiently; the decomposed form fragments.

This matters more than it sounds for two reasons. First, cost: text arriving in decomposed form can be noticeably more expensive for the same content. Second, caching: a prompt prefix that is byte-identical caches, and one that differs by normalisation form does not, even though it looks the same on screen. Prompt caching explained covers how strictly that prefix match is enforced.

Normalising input to a single form before it reaches the model removes both problems and costs almost nothing.

Invisible characters cause invisible bugs

Zero-width joiners, zero-width spaces, byte order marks, non-breaking spaces and directional marks all occupy bytes and therefore tokens while displaying as nothing or as an ordinary space.

Text pasted from a word processor, a PDF or a web page routinely carries these. They inflate token counts, they break exact-match caching, and they can shift a model's behaviour by fragmenting words that would otherwise tokenise cleanly.

They are also a security surface. Invisible characters can hide instructions inside text that looks innocuous to a human reviewer, which is a real vector when your pipeline feeds retrieved or user-supplied documents into a prompt. Prompt injection and agent security covers the broader pattern.

Stripping control and formatting characters that your application has no use for is a small piece of hygiene with outsized returns.

Truncation is where it actually breaks

The most common production failure is cutting text to fit a budget and cutting through a character.

Slicing a UTF-8 byte string at an arbitrary offset can land in the middle of a multi-byte sequence, producing an invalid fragment that renders as a replacement glyph or throws when decoded. Slicing by code point can split a grapheme cluster, leaving a combining accent orphaned or a joined emoji broken into parts.

Slicing by token is the version that matters for models, and it has its own hazard: a token boundary is not a character boundary, so a token-level cut can also produce a broken character at the seam.

The safe pattern is to truncate at a semantic boundary you choose — a sentence, a paragraph, a document, a chunk — and then verify the result fits the token budget, rather than cutting at exactly the budget. Losing a sentence is cheap; emitting invalid text into a prompt is not.

Where the cost lands unevenly

Because tokens track bytes, and bytes track script, the same meaning costs different amounts in different languages. Latin-script text is cheapest, since it dominates training corpora and encodes in one byte per character. Scripts using two or three bytes per character and appearing less often in training pay twice.

This is not a policy choice by any provider; it falls out of byte-level tokenisation trained on a web-scale corpus. It is still a real budget line if your users write in those scripts. Token costs by language covers how to measure your own multiplier instead of guessing.

A short checklist

Normalise incoming text to one Unicode form at the edge of your system. Strip control and zero-width characters you do not need. Count tokens with the real tokenizer rather than estimating from character length, because the estimate is wrong by a factor that depends on the script.

Truncate at semantic boundaries and check the token count afterwards. And if you are budgeting for a multilingual product, measure the per-language multiplier on your own traffic before you set a price. Reducing token usage covers what to do once you know where the tokens are going.