Tokenization: How Text Becomes Tokens, and Why It Matters

What a token actually is

A language model never reads letters, and it does not quite read words either. Before your text reaches the model, a tokenizer converts it into a sequence of tokens: subword units drawn from a fixed vocabulary that was learned before the model was trained, typically tens of thousands to a few hundred thousand entries. Each token maps to an integer id. The model consumes a list of ids, predicts the next id, and a decoder turns ids back into text at the very end. Everything in between, all the attention and computation described in the transformer deep-dive, happens over ids, not characters.

Tokens sit at an awkward granularity that takes a moment to internalize. Common English words usually map to a single token. Rarer words split into two or more pieces. Whitespace and punctuation get folded in, often attached to the front of the following word. A single short sentence might split like this:

Input:   "Tokenization is unavoidable."

Pieces:  | Token | ization |  is |  un | avoid | able |  . |
Ids:     | 28714 |    1634 | 318 | 555 |  9765 |  616 | 13 |

Notice what happened: a word the vocabulary knows well survives intact, while a longer derived form breaks into recognizable fragments. The model then learns, during training, how these fragments behave statistically in context. It has rich knowledge about how ids co-occur, but no direct view of the letters inside them.

That single design decision is the root of everything else in this article. Billing surprises, context limits, multilingual cost gaps, and the famous failures at letter-counting all trace back to the fact that the model's atomic unit of perception is a token id, not a character.

Why subwords won

Subword tokenization is a compromise between two simpler schemes that both fail at scale. Working at the character or byte level gives you a tiny, complete vocabulary, but every letter becomes a sequence position: a one-page document turns into thousands of steps, sequences get long, and the model spends capacity relearning spelling before it can learn meaning. Working at the whole-word level keeps sequences short, but the vocabulary is unbounded. Every name, typo, product code, and newly coined term is a word the tokenizer has never seen, and a word-level system has no way to represent it at all.

Subwords take the middle path. Frequent strings, whole common words included, earn their own vocabulary entry, so ordinary prose stays compact. Rare words decompose into pieces the model does know: an unfamiliar drug name or a fresh internal codename becomes three or four fragments rather than an unrepresentable blank. And because the vocabulary bottoms out at bytes or characters, nothing is ever truly out of vocabulary; the worst case is just an inefficient split.

The result balances two budgets at once: vocabulary size, which the model must embed and predict over, against sequence length, which drives compute and context consumption. Every serious model family made the same call, which tells you how decisively this trade resolved.

How BPE-style tokenizers are built

The dominant recipe is byte-pair encoding, or BPE, and it is refreshingly mechanical. Start with a corpus of training text represented as bytes or characters. Count every adjacent pair of symbols. Merge the most frequent pair into a single new symbol, add it to the vocabulary, and recount. Repeat, thousands of times, until the vocabulary reaches a target size. The learned merge list is the tokenizer: at inference time, incoming text is split by replaying those merges in order.

Corpus keeps repeating "token", "tokens", "tokenizer"...

Start:    t  o  k  e  n  s
Merge 1:  (t,o)   -> to      to  k  e  n  s
Merge 2:  (to,k)  -> tok     tok  e  n  s
Merge 3:  (e,n)   -> en      tok  en  s
Merge 4:  (tok,en)-> token   token  s

After enough merges, "token" is one vocabulary entry
and "tokens" encodes as two: [token][s].

Two properties fall out of this procedure. First, the vocabulary is a mirror of the training corpus: whatever appeared often gets its own compact token, and whatever appeared rarely stays fragmented. Since these corpora are typically dominated by English and by code, English and code tokenize efficiently, and much else does not. Second, the tokenizer is a frozen artifact. It is trained once, before the model, and shipped alongside it; the model's entire understanding of text is filtered through that one fixed split for its whole life.

Variants exist. SentencePiece-style toolkits handle raw text without assuming spaces separate words, and unigram-model approaches select a vocabulary probabilistically rather than by greedy merging. The differences matter to tokenizer builders, but the enterprise-relevant behavior is the same everywhere: a learned, frozen, corpus-shaped vocabulary of subword pieces.

Where tokenization bites

The first practical consequence: token counts are not portable across providers. Every model family trains its own tokenizer on its own corpus, so the same contract, prompt, or knowledge-base article produces different counts on different models. A capacity estimate or budget expressed in tokens is meaningless until you say whose tokens.

The second is a genuine cost inequity: non-English text usually needs more tokens per sentence. Vocabularies skew toward the languages that dominate training corpora, so a sentence in German or Italian tends to fragment more than its English translation, and non-Latin scripts often fare worse still, sometimes by a substantial multiple. Same meaning, same model, higher bill and less usable context. If you operate multilingual workloads, this belongs in your cost model, not in a footnote.

Third, numbers, code, and whitespace split in unintuitive ways. A long number may break at arbitrary digit boundaries, which is one reason models are shaky at arithmetic on large figures. Code indentation, JSON punctuation, and runs of spaces each consume tokens in ways that are hard to eyeball. Two files of identical character length can differ meaningfully in token count.

None of these quirks are bugs to be patched. They are structural properties of subword tokenization, and they persist across model generations, so design around them rather than waiting for them to disappear.

Tokens are the billing meter

API pricing is denominated per token, with separate rates for input and output. That makes the tokenizer itself a line item: for identical text, a model whose tokenizer produces fewer tokens is charging you for less metered volume. Your effective cost is always the per-token price multiplied by the token count your content actually produces, and both factors vary by model.

This is why per-token price comparisons mislead. The same 20-page document might tokenize to noticeably different counts on two providers, and the multilingual penalty from the previous section compounds this: a workload that is 40 percent non-English can rank two models differently than their headline prices suggest. The full cost picture, including output pricing and caching, is covered in the LLM economics deep-dive, and you can pressure-test scenarios in the cost estimator in the Lab.

For quick English-language estimation, a serviceable rule of thumb is roughly three to four characters per token, or about three-quarters of a word. Treat it as a sizing aid, not a number to put in a business case: it drifts with formatting, jargon, code content, and language mix. The moment real money is attached, count rather than estimate; every provider exposes token counts in API responses, and most offer counting endpoints or libraries.

Tokens are the context budget

The context window, the total amount of input and output a model can attend to at once, is measured in tokens too. So tokenization does not just set your price; it sets your capacity. Two models with the same advertised window hold different amounts of your text, because their tokenizers compress it differently. How windows behave at scale, including why bigger is not automatically better, is the subject of the context windows deep-dive; here the point is simply that the window is spent in tokens, not pages.

In practice, the budget disappears faster than character counts suggest, and a few offenders account for most of the loss:

• Verbose serialization. JSON and XML repeat keys, quotes, brackets, and tags on every record; the structural overhead can rival the payload. Compact formats such as minified JSON, CSV, or plain prose often carry the same information in far fewer tokens.
• Boilerplate at scale. Log timestamps, email signatures, legal footers, and repeated headers are cheap to a human skimmer and expensive to a token meter.
• Inefficiently tokenized languages. The multilingual penalty applies to the window exactly as it applies to the bill: the same document in a non-Latin script may fit only a fraction as much content into the same window.

This is why prompt and pipeline design is partly a compression exercise. Stripping boilerplate before retrieval, choosing lean output schemas, and summarizing history instead of replaying it are all token-budget decisions, and teams that make them deliberately fit dramatically more useful signal into the same window than teams that paste and pray.

The architect view

Tokenization is one of those foundations that looks like trivia until it shows up in a budget review or an incident channel. The architect's job is not to build tokenizers; it is to stop being surprised by them. Three habits cover most of that ground.

1. Know whose tokens you are counting. Every estimate of cost or capacity is denominated in a specific model's tokenizer. When you switch models, recount; when you compare models, normalize to cost per document or per transaction, never per token.
2. Measure on your own corpus. Rules of thumb calibrated on English prose fail quietly on codebases, contracts, logs, and multilingual content. Before committing numbers to a business case, run the candidate tokenizers over a representative sample of the real workload. It is an afternoon of work that regularly moves estimates by double-digit percentages.
3. Remember the input resolution. When a model fails at counting letters, validating checksums, or manipulating exact character positions, the diagnosis is usually tokenization, not intelligence. Route character-level operations to code and keep the model on token-level work, where it is strong.

There is also a quiet strategic point here. Token efficiency is a lever you control: format choices, boilerplate stripping, language-aware routing, and schema design all change the denominator of every per-token price you pay. Most organizations negotiate hard on rates and never touch the counts. The counts are usually the easier win.