Every number attached to a large language model is denominated in the same strange currency: tokens. Context windows are 200,000 tokens. APIs bill per million tokens. Rate limits count tokens per minute. And yet a token is not a word, not a character, and not the same thing from one model to the next. The mismatch explains a lot of confusing behavior, from surprise invoices to ChatGPT failing to count the letters in "strawberry."
The model never sees letters
A language model does not read text. It reads a list of integers. Before a single layer of any transformer runs, a tokenizer splits your prompt into chunks, looks each chunk up in a fixed table called the vocabulary, and replaces it with that chunk's number. The model then works exclusively with those numbers, and its output is one number at a time, decoded back into text for you.

Take the word "strawberry." In OpenAI's cl100k vocabulary, the one behind GPT-3.5 and GPT-4, the bare word splits into three chunks: "str," "aw," and "berry," token IDs 496, 675 and 15717. In o200k, the newer GPT-4o-era vocabulary, it splits differently: "st," "raw," and "berry," IDs 302, 1618 and 19772. Neither split is "straw" plus "berry," the division any human would pick. And inside a sentence, with a leading space attached, " strawberry" collapses into a single ID in both vocabularies: 73700 in cl100k, 101830 in o200k. One indivisible number.
That is why models famously failed the letter-counting question. The model was never handed ten letters. It was handed one number, and the letters inside that number are invisible to it. Later reasoning models got better at the task by spelling the word out as separate letter-tokens before answering, turning the chunk back into something they can count. Microsoft's developer docs describe the same mechanics in general terms: tokenization runs as the first step, each unique token gets an ID, and the model reasons over IDs, not strings.
Why not just use words, or letters?
Tokenization is a compromise between two bad options.
Letter-by-letter (or byte-by-byte) input gives universal coverage: every word in every language, plus every typo, fits in the vocabulary because the vocabulary is just 256 bytes. The cost is sequence length. A 100-word paragraph becomes 500-plus steps, and transformers pay quadratically in sequence length, so training and inference get expensive and the model must spend capacity stitching letters into meaning.
Word-by-word input is compact and readable, but the vocabulary would need millions of entries to cover inflections, misspellings and rare terms, and any unseen word is unrepresentable.
Subword tokenization splits the difference: common words and common word-fragments each get their own entry, rare words decompose into a few pieces. Modern vocabularies are built by byte pair encoding, which merges the most frequent adjacent byte pairs in a training corpus until the vocabulary reaches a target size. That construction process deserves its own article, and we have one; the point here is the consequence: the vocabulary is a fixed table chosen before training, and its size is a design decision, not a law of nature.
The tables are not small, and they are not comparable. Meta moved Llama 3 from a 32,000-entry SentencePiece vocabulary to a 128,000-entry tiktoken one, which encoded the same text in fewer tokens and improved coding and reasoning benchmarks along the way. A 2026 audit by ellamind tested 20 current tokenizers and found vocabulary sizes spanning roughly 16,000 to 262,144 entries: Llama 3.x at 128,256, OpenAI's o200k at 201,088, Alibaba's Qwen 3.8 at 248,077, and Google's Gemini 3.x/Gemma 4 tokenizer at 262,144. The same sentence therefore occupies a different amount of "model space" depending on which model reads it.
Your prompt is a different text on every model
Because each provider learned its own vocabulary on its own corpus, token counts diverge on the same input. Researchers at Cambridge and the University of Hong Kong tested ten tokenizers against single words and found "antidisestablishmentarianism" comes out at 4 to 9 tokens depending on the model: 9 on one Claude tokenizer, 5 on Gemini and Grok. Their verdict on the popular rule of thumb that one token equals roughly four English characters: "inadequate and oversimplified." On randomly sampled English words they measured 0.35 to 0.45 words per token, against the 0.75 the rule promises.
Tokenizers also drift inside a provider's own lineup, and that is where bills move. Anthropic introduced a new tokenizer with Claude Opus 4.7 in 2026 and states in its pricing documentation that the tokenizer "produces approximately 30 percent more tokens for the same text than earlier models," with a range of 1x to 1.35x depending on content. The ellamind audit found the effect larger on its multilingual corpus: about 65% more tokens on average than the reference tokenizer, and roughly double on German text. Two practical consequences Anthropic's own migration guide spells out: fixed max_tokens caps can start truncating output after such a switch, and a lower per-token price does not guarantee a cheaper bill. If the token count rises 30% and the price falls 20%, the cost goes up about 4%.
The tokenizer tax on languages

A vocabulary is a record of what its creators read. Frequent strings earn short, whole-word entries; everything else fragments. That gives English an unfair head start, and it converts directly into cost.
Ahia and colleagues' widely cited study of ChatGPT-era tokenizers found that among supported languages, some require up to five times as many tokens as others to convey the same information, with own-script languages like Telugu and Georgian paying the most. A 2026 controlled study across ten foundation models and 25 European languages measured the spread precisely: English runs about 1.2 tokens per word, Romance languages 1.5 to 1.7, Germanic 1.7 to 1.9, Slavic 2.2 to 2.5, and Greek and Maltese top out around 3.1. The same API call expressing the same request costs two and a half times more for a Greek user than an English one. Vocabulary design can bend the curve: Chinese costs roughly 15% more tokens than English under OpenAI's cl100k vocabulary, but less than English under Zhipu's GLM tokenizer, which was trained with more Chinese text.
Where tokens show up in your work
- Price. APIs bill input and output tokens separately, per million. As of late 2026, Anthropic lists Claude Sonnet 5.5 at $2 per million input and $10 per million output, and Haiku 5.5 from $0.10 and $0.50. Any per-token price comparison across providers is comparing different sized chunks, which is why a "cheaper" model can still cost more on your traffic.
- Context windows. Windows are specified in tokens, and tokens are not interchangeable across vocabularies. Anthropic lists 1M tokens on its current models, and its own documentation estimates that window at roughly 555,000 to 750,000 words of English depending on the tokenizer generation in use.
- Latency and budgets. Models decode one token per step. A tokenizer that needs 30% more tokens to say the same thing makes every answer 30% longer to stream, and reasoning models that "think" in tokens hit their thinking budgets sooner.
- Counting. Never count tokens with a character heuristic. Use the provider's own counter, Anthropic's
count_tokensendpoint or OpenAI's open-source tiktoken library, and re-count after any model migration, because the number changes even when your prompt does not.
The token is the one artifact of 2010s-era tokenizer engineering that still sits between you and every model, deciding what it can see, what it costs, and how fast it can speak. It is worth knowing what the unit actually is before trusting any number expressed in it.
Sources
- Microsoft Learn: Understanding tokens in LLM APIs - https://learn.microsoft.com/en-us/dotnet/ai/conceptual/understanding-tokens
- MemX: Why AI Can't Count R's in Strawberry (tiktoken IDs) - https://memx.app/blog/why-ai-cant-count-rs-in-strawberry
- ellamind: Tokenization Tax 2026, LLM tokenizer comparison - https://www.ellamind.com/blog/tokenization-tax-report-2026
- Firecrawl: AI Tokens, what they are, how they're counted (Cambridge/HKU tokenizer study) - https://www.firecrawl.dev/blog/llm-tokenization-across-models
- Anthropic: Models overview and Pricing documentation - https://platform.claude.com/docs/en/about-claude/models/overview - https://platform.claude.com/docs/en/about-claude/pricing
- Anthropic: Token counting - https://platform.claude.com/docs/en/build-with-claude/token-counting
- Developers Digest: The Claude Tokenizer Change - https://www.developersdigest.tech/blog/claude-tokenizer-change-cost-impact
- Ahia et al.: Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models - https://orevaahia.github.io/assets/pubs/Do_All_Languages_Cost_the_Same.pdf
- arXiv: The Tokenizer Tax Across 24 European Languages - https://arxiv.org/html/2605.24718
- arXiv: Mythbuster: Chinese Language Is Not More Efficient Than English (tokenizer vocabulary allocation) - https://arxiv.org/html/2604.14210v1
- Meta / Llama: Understanding the Llama 3 Tokenizer (128k vocabulary, tiktoken) - https://www.youtube.com/watch?v=Tmdk_H2WDj4
- Hugging Face LLM Course: BPE tokenization - https://huggingface.co/learn/llm-course/en/chapter6/5
