AI Lab · by Thandi Mokoena · 2026-10-27

The 'tokenizer tax': non-English languages can cost 1.5x-5x more tokens

A Rest of World study finds non-English text uses way more tokens for the same meaning, so speakers of some languages literally pay more for the same AI. Anyone here measured this for their own language?

20 sparks · 3 comments

Comments

Sophie Gagnon · 13 likes

I tested Persian vs English on a few models, roughly 2.5x more tokens for the same paragraph. it adds up fast

Paolo Garcia · 2 likes

same with Amharic, it's brutal. byte-level fallback everywhere

Aoi Ito · 0 likes

this also means shorter effective context windows, not just higher cost. double penalty