Using the same AI model in Amharic, Burmese or Khmer can cost several times more than using it in English, and a new study puts numbers on the gap for 19 model families. The Coalition for Language Equity in AI Research (CLEAR) calls it the "tokenizer tax." Rest of World reported on the study on October 26, and CLEAR found that no general-purpose frontier model in its test earns better than a B on its grading scale.
What a tokenizer tax is
Language models do not read words. A tokenizer first splits text into chunks called tokens, and AI providers typically bill by the token. Our byte-pair encoding explainer shows how many tokenizers build that vocabulary. CLEAR's point is that tokenizers learn mostly from English and a few other languages with a large online presence, so text in other languages gets cut into smaller pieces and needs more tokens for the same meaning.
CLEAR's own example uses one sentence: "People should benefit from AI, regardless of the language they speak." With the GPT-5 tokenizer, CLEAR counts 13 tokens in English, 26 in Malayalam and 78 in Amharic, where each Ge'ez character is split into two single-byte tokens. "That makes providing input to GPT-5 in Amharic six times more expensive than English," the CLEAR authors write.
CLEAR also addresses the obvious objection, that some languages are simply longer. It says that effect "is measured in tens of percent." Shan, the most verbose language in its dataset, is only 42% longer than English at the character level, according to the study, which is far smaller than the gaps it measured.
How CLEAR measured it
The authors, Noah Perry, Anjali Kantharuban, Sarah Luger and Francisco Guzmán, published the study on September 29, 2026. They ran the tokenizers of models from major labs over FLORES-200, a set of sentences professionally translated into about 200 languages. Their headline metric is the max/min ratio: how many times more tokens the most expensive language needs than the cheapest one for the same content. They also report a Gini coefficient, the inequality measure economists use for income.
The grading scale is strict on purpose. A ratio of 3x or less earns an A, up to 6x a B, up to 9x a C, up to 12x a D, and anything above 12x an F. CLEAR says the metric "captures what the worst-served language pays because that's the language whose speakers are priced out first."
The results
Among the top 20 languages by web presence, the models are close to even. CLEAR reports ratios between 1.5x and 3.1x there "with only one exception," and its table lists Falcon 3 from TII at 4.5x. Across all of FLORES-200, the spread widens to between 2.7x and 17x.
Selected grades from CLEAR's table, by FLORES-200 max/min ratio:
- Meta NLLB-200, a translation model: 2.7x, grade A
- Google Gemini and Gemma family: 5.1x, grade B
- Anthropic Claude Fable 5 and Opus 4.8: 9.3x, grade D
- Meta Llama 4 Scout and Muse Glimmer: 9.5x, grade D
- DeepSeek V4.1 Flash: 10.6x, grade D
- OpenAI GPT-5 and gpt-oss: 13.8x, grade F
- Mistral Ministral 3 and NVIDIA Nemotron 3.5: 14.9x each, grade F
- TII Falcon 3: 17.0x, grade F
CLEAR notes NLLB is not a fair bar for a frontier model, because it does not have to handle code, math or English at frontier quality. It treats NLLB as proof of what a vocabulary built for many languages can do, and Google's family as proof that a general-purpose model "can get most of the way there."
A worst-case ratio does not tell the whole story, and CLEAR says so. On a second measure, the share of languages that need less than twice the tokens of the model's best language, CLEAR names GPT-5 the best performing model, at 158 languages, or 78%. Only the NLLB translation model scores higher in its table, at 194 languages (96%). CLEAR says high max/min ratios like GPT-5's can be driven by a small number of languages that the tokenizer handles very poorly.
CLEAR reports two other findings. Newer is not always better: CLEAR reports that Alibaba's Qwen 3.6 scores 13.1x, worse than the older Qwen 3 at 10.5x, while Meta improved from 14.8x on Llama 3.3 to 9.5x on Llama 4. And the problem is "largely driven by the script," CLEAR says. Asturian, a Latin-script language with few speakers, does well, while Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki were at the biggest disadvantage "across nearly all models in our analysis," according to the study. CLEAR adds that large languages pay too, naming Punjabi, Bengali and Urdu, each with hundreds of millions of speakers. Our explainer on the curse of multilinguality covers tokenizer fertility, the average number of tokens per word, and why it varies by language.
Why cost is only part of it
More tokens also means slower answers and less room in a model's context window. "A nominal 128k-token context window can only hold a fraction of the content in a heavily taxed language, putting long-document and agentic use cases out of reach," CLEAR writes. For a sense of scale on price, CLEAR cites Anthropic's Fable 5 at $10 per million input tokens and $50 per million output tokens, and notes that providers typically charge by the token "irrespective of language." Our LLM API pricing comparison covers list prices by provider.
Rest of World asked AI founders and investors about the gap. Pratyush Choudhury, co-founder of the AI venture fund Activate AI, compared it to a tutor who must remember a student's earlier mistakes: "If that history consumes the context window much faster in one language, the disadvantage reaches into the quality of the service," Choudhury said, according to Rest of World. Dheemanth Reddy, CEO of Maya Research, expects the price impact to shrink as token prices fall, but said fixing the gap "will still require labs to deliberately design future tokenizers to handle these languages and scripts better."
What CLEAR asks labs to do
CLEAR makes three requests. Labs should publish the FLORES-200 max/min ratio and the share of languages under 2x in model cards. They should set vocabulary targets for the next generation of tokenizers. And they should change pricing now, since a tokenizer cannot change until the next base model is trained. "Character-normalized or language-normalized pricing would eliminate the tokenizer tax immediately," the authors write, adding that this helps only for models sold per token through an API.
CLEAR also says governments buying AI for multilingual populations should require token cost disclosure. Its code is on GitHub, and it says a companion study on quality across languages is underway. Neither CLEAR nor Rest of World published a response from the labs named.
