A study released last month by the Coalition for Language Equity in AI Research reveals that current AI tokenizers treat languages unevenly, creating a hidden “tokenizer tax.” Tokens, the basic units processed by language models, are consumed at different rates depending on the language, which can inflate computational costs for speakers of less-represented tongues.
Within the twenty most common internet languages, the analysis found the most token-hungry language used between one-and-a-half and five times more tokens than the cheapest option. When the researchers broadened the sample to two hundred languages, the disparity widened dramatically, with one model’s token cost gap expanding by a factor of seventeen.
Because token limits define how much text a model can retain, users of non-English languages lose more of their document or dialogue history, and the system must expend additional effort to produce answers. “Think about a tutor: retaining the student’s earlier mistakes and explanations is part of delivering a good lesson,” explained Pratyush Choudhury, co-founder of Activate AI, to Rest of World.
Open-data projects aim to narrow the gap by expanding high-quality corpora for under-served languages. Raoul Nanavati, co-founder and CEO of Navana.ai, told Rest of World that the RESPIN initiative, a partnership with the Indian Institute of Science funded by the Gates Foundation, has gathered native-speaker text and read speech from thirty-eight dialects, providing valuable material for future tokenizers.
The research also highlighted that a language’s script heavily influences token consumption. Asturian, despite its modest speaker base and limited web presence, benefits from the Latin alphabet and similarity to other Romance languages, reducing its token load. Conversely, languages written in scripts rarely encountered by tokenizers,such as Burmese, Tibetan, Tifinagh, Khmer, Lao, and Ol Chiki,consistently incurred the highest token penalties across the models examined.
Industry leaders acknowledge that token prices are dropping quickly, which should lessen immediate financial strain, yet they stress that the underlying bias will persist without intentional redesign. Dheemanth Reddy, CEO of Maya Research, warned to Rest of World that synthetic multilingual data alone cannot resolve the tokenizer gap, and that labs must purposefully engineer tokenizers for diverse scripts. Choudhury added that language efficiency ought to be a core product metric alongside accuracy, speed and reliability.