Every LLM API pricing table rests on an unstated assumption: that a token is a standard unit, like a kilowatt-hour or a gigabyte. It is not. A token is a unit of a vendor's own choosing, and two providers can look at the same sentence and disagree about how many units it contains. The disagreement is not small, and it is not random. It is worth tens of percent on ordinary English text and several multiples on structured content. That means the headline "$ per million tokens" numbers that anchor almost every model comparison are not directly comparable, and the cheapest model in the table is often not the cheapest bill.
What the measurements show
The cleanest public comparison of this comes from TensorZero, which in April 2026 sent identical inputs through OpenAI, Anthropic and Google's official token counting APIs and normalized the results against OpenAI's count. The multipliers, from their post Stop comparing price per million tokens:
| Model | Plain text | YAML | JSON | Tool definitions |
|---|---|---|---|---|
| gpt-5.4 | 1.00x | 1.00x | 1.00x | 1.00x |
| gemini-3.1-pro | 1.06x | 1.18x | 1.11x | 1.82x |
| claude-sonnet-4-6 | 1.17x | 1.25x | 1.22x | 2.06x |
| claude-opus-4-6 | 1.17x | 1.25x | 1.22x | 2.06x |
| claude-opus-4-7 | 1.57x | 1.53x | 1.70x | 2.65x |
The inputs were the text of the Iliad, a trimmed Cloudflare OpenAPI spec as JSON and YAML, and 100 generated tool schemas. Multiply each model's list price by its multiplier and the comparison tables fall apart. gpt-5.4 lists at $2.50 per million input tokens and claude-opus-4-7 at $5.00, so the price list says Opus costs twice as much. On tool definitions, after the tokenizer counts are applied, Opus 4.7 costs the equivalent of $13.25 per million of OpenAI's tokens, 5.3 times gpt-5.4 for identical bytes. Gemini looks cheapest on plain text ($2.12 effective against OpenAI's $2.50) yet becomes more expensive than OpenAI on tool schemas. The ranking depends on what you send.

The vendors say it themselves
Anthropic's platform pricing documentation is unusually direct about the problem: "Claude 4.7 and later models ... use a newer tokenizer that contributes to their improved performance on a wide range of tasks. This tokenizer produces approximately 30% more tokens for the same text." Claude Opus 4.7 and Opus 4.6 share the same $5/$25 rates, which means the newer model bills the same fixed text for roughly 30 percent more, a price increase that never appears in any announcement. Third-party cost analyses of the change put the practical consequence bluntly: a model migration with a 20 percent rate cut and a 30 percent token increase raises the bill by about 4 percent, and per Developers Digest's re-check of Anthropic's docs, the wording previously read "up to 35% more tokens" before settling at "approximately 30%."
The reason labs change tokenizers at all is performance, not billing. A larger or better-segmented vocabulary compresses text into longer token runs per word in some languages but improves multilingual coverage and model behavior. Anthropic's own note says the new tokenizer "contributes to their improved performance." The cost effect is the byproduct nobody reprices.
Why the counts diverge
Modern LLM tokenizers are almost all byte-pair encoding systems, and each lab trains its vocabulary on its own corpus, to its own target size, with its own merge rules. There is no cross-vendor standard. A 2026 tokenizer comparison by ellamind measured 20 tokenizers on 21 kinds of text in 12 languages, normalizing every count against OpenAI's o200k encoding. Its summary: Cohere's Command A+, Meta's Llama-4-derived vocab, xAI's Grok and Alibaba's Qwen tokenize efficiently across the board, while the Claude tax runs +65 percent more tokens than the o200k reference on average, and +101 percent on German text. The same report documents how narrow some vocabularies are: reconstructed Claude 4.7+ vocabularies sit near 16,000 entries, against 262,144 for Gemini 3.x. Fewer entries means more pieces per word.
The language dimension has academic backing that predates the current pricing debate: Petrov et al. at NeurIPS 2023 measured token cost per character across 100-plus languages and found speakers of Burmese, Tamil and Malayalam paid 5 to 15 times more tokens per character than English speakers under 2023-era GPT, Llama and BERT tokenizers. Per-token prices are not just vendor-local. They are vendor- and reader-local.
What else the mismatch distorts
The billing is the least of it. Tokenizer efficiency also breaks three other comparisons engineers make routinely, as ellamind's report lays out:
- Context windows. A 1 million token window holds a different amount of text per provider. Claude's window and Gemini's window are not the same size in words.
- Throughput benchmarks. Tokens per second measured on a model with an inefficient tokenizer is a smaller number for the same text, independent of hardware.
- Reasoning budgets. Output caps and thinking budgets are denominated in tokens, so a model whose tokenizer needs more tokens to express a thought gets less reasoning done before hitting the cap.
There is also a middle-layer confusion: the same model's counts differ depending on which API surface you call. Developers testing Gemini through two different SDKs reported counts of roughly 1,300 versus 3,000 for the same request, because wrappers add hidden chat-template and image-encoding overhead before the model proper sees anything.
What to do instead
Two rules follow from the measurements.
Count with the vendor's counter, never your own. OpenAI's o200k encoding runs locally and exactly via tiktoken. Anthropic exposes a token counting endpoint, Google a countTokens API, and xAI's Grok API a tokenize_text function that returns the actual split. Word-count heuristics and "100 words is roughly 130 tokens" rules of thumb are estimates in one vendor's unit applied to another vendor's bill.
Price tasks, not tokens. The practical metric is cost per completed task on your content: take real request logs, count them through each shortlisted provider's counter, apply each provider's cache-read and long-context tier rules, and compare totals. Octomind's October 2026 pricing comparison does exactly this and finds the same reversal TensorZero predicts: models with identical per-token lists land 30 to 50 percent apart per task once tokenizer counts, cache rates and context tiers are applied. If you are migrating between model versions within one vendor, recount everything first. Anthropic's own token counting docs tell you to count against the model you plan to use rather than reuse counts from an older one.
The uncomfortable summary: the industry's shared price currency is not shared. "A million tokens" is a quantity you can only define by naming a vendor. Until the labs converge on a unit, the only honest comparison table is the one built from your own prompts, counted twice.
Sources
- TensorZero: Stop comparing price per million tokens: the hidden LLM API costs
- Anthropic: Platform pricing documentation and Token counting guide
- ellamind: Tokenization Tax Report 2026
- Petrov et al.: Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023)
- Octomind: LLM API Pricing Comparison 2026: What a Task Costs, Not a Token
- safeguard.sh: Model migration cost: when a new tokenizer changes your bill
- Developers Digest: The Claude Tokenizer Change: What ~30% More Tokens Means for Your Bill
- Google AI Developers Forum: Token count differences between SDKs for Gemini
