What does BPE actually do during training?

It starts from raw bytes, counts every adjacent pair across a corpus, merges the most frequent pair into a new vocabulary entry, and repeats until the vocabulary reaches its target size. The output is an ordered list of merge rules.

Why can models not count letters in a word?

The model never sees letters. BPE hands it a token standing for a whole chunk of characters, so individual letters inside that chunk were never separate inputs. Give the model a tool for character-level work instead.

Are token counts the same across models?

No. Each model trains its own merge rules on its own corpus, so the same text can cost meaningfully different token counts across providers. Always count with the tokenizer of the model you are billing against.

Byte-Pair Encoding Explained: How Tokenizers Are Built

Byte-pair encoding was invented as a data compression method and repurposed for language models because it solves a problem neither words nor characters solve. Nearly every model you use is tokenised by BPE or a close relative, and its quirks explain several behaviours that otherwise look like bugs.

The problem it solves

You could split text into words. That gives short sequences and readable units, but the vocabulary is unbounded — every typo, identifier, hashtag and inflected form is a new word — and anything unseen becomes an unknown token the model cannot represent at all.

You could split into characters instead. The vocabulary is tiny and nothing is ever unknown, but sequences become several times longer, and since attention cost grows with the square of sequence length, that is expensive in exactly the wrong way. Attention mechanisms explained covers why that scaling dominates.

BPE sits in between. Common sequences become single tokens, rare sequences decompose into smaller pieces, and because the base units are bytes, nothing is ever unrepresentable. You get short sequences for ordinary text and graceful degradation for everything else.

How the vocabulary is trained

BPE is trained on a corpus before the model is trained, and the procedure is mechanical.

Start with every text split into its raw bytes. Count every adjacent pair of units across the corpus. Find the most frequent pair and merge it into a single new unit, recording that merge as a rule. Recount, merge the next most frequent pair, and repeat.

Each iteration adds exactly one entry to the vocabulary. You stop when the vocabulary reaches its target size — commonly somewhere between fifty thousand and a few hundred thousand entries in current models.

What comes out is an ordered list of merge rules. Encoding new text means applying those rules in the same order until no more apply. That determinism matters: the same string always produces the same tokens, which is what makes prompt caching over a stable prefix work at all.

What the resulting vocabulary looks like

The merges follow frequency, so the vocabulary ends up mirroring the training corpus rather than any linguistic theory.

Very common English words become single tokens. So do common word fragments, common punctuation sequences, and — importantly for code — leading whitespace runs, so that four spaces of indentation is often one token rather than four.

Frequency also means position matters. A word at the start of a sentence and the same word preceded by a space are frequently different tokens, because BPE learned the space-prefixed form as its own unit. This is why token counts change slightly when you reflow a prompt, and why leading whitespace in your API calls is not free.

Rare material fragments. An unusual identifier, a long hexadecimal string or a chemical name decomposes into many small pieces, because no merge rule ever fired for those sequences during training.

Why BPE causes the strange failures

Several well-known model weaknesses trace directly back to this.

Counting letters inside a word is hard because the model does not see letters. It sees a token that stands for a whole chunk, and the individual characters inside it were never separate inputs. Asking how many times a letter appears in a word is asking about something below the model's perceptual resolution.

Arithmetic on long numbers is unreliable for the same reason. Digit sequences are grouped by whatever the merge rules happened to produce, so the same numeric value can tokenise inconsistently depending on its surroundings, and column alignment is not visible to the model.

Rare-token behaviour is the third case. Tokens that appeared very seldom in training have poorly trained representations, and prompts that land on them can produce erratic output. Tokenization explained goes through the practical bugs this produces.

Byte-level BPE and why it matters

Early BPE implementations operated on Unicode characters and needed an explicit unknown token for anything outside the training alphabet. Byte-level BPE removed that by starting from the 256 possible byte values instead.

Since any text encodes to bytes, every possible input is representable. Emoji, unusual scripts, mixed encodings and binary-looking data all tokenise without failure — they simply tokenise inefficiently, consuming several tokens where familiar text would consume one.

That inefficiency is the cost you pay for universality, and it falls unevenly. Text in scripts underrepresented in the training corpus decomposes into far more tokens per unit of meaning than English does, which turns directly into a price difference. Token costs by language covers the size of that gap.

Variants you will encounter

Most production tokenizers are BPE or something adjacent, with differences that matter less than the family resemblance.

WordPiece, used in the BERT lineage, merges by a likelihood criterion rather than raw frequency. Unigram, used by SentencePiece, works the other direction — it starts with a large candidate vocabulary and prunes pieces that contribute least, then picks the most probable segmentation of a given string.

SentencePiece is best understood as a wrapper that treats the input as a raw stream including spaces, so it does not depend on language-specific pre-tokenisation rules. That makes it the common choice for multilingual models.

The practical consequence of all this variety is that token counts are not portable between models. Two models can bill for the same prompt at meaningfully different token counts, which is why cost comparisons should be run on your own text. Tokenizer differences across models covers how large that spread gets.

What to do with this

Count tokens with the tokenizer of the model you are actually calling, never with a character-based estimate and never with another model's tokenizer. The estimate is wrong in a direction that depends on your content.

If a task involves characters, digits or precise spelling, do not fight the tokenizer — give the model a tool. Character counting, arithmetic and string manipulation are cheap and exact in code and unreliable in a language model, and the failure is architectural rather than a matter of prompting harder.