Posted by DeepLogin 4 days ago
Because, right now the main issue is that some languages are tokenized much more efficiently than others and it is a zero sum game, since there are a limited number of tokens and by adding tokens that are needed for one language you will inevitably have to remove tokens for another language and therefore decrease efficiency for it.
[0] https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1
However, they (Cartesia) haven't yet published a technical paper on their new version of H-Net, and it is unclear if they will.
[1] https://lifthrasiir.github.io/aqueous/ (specifically wrt-recons-hangul codec)
Even if you use bytes only, Unicode is biased towards English and other Latin languages, too.
Also, it is non-EU, so initiatives like [0] will only indirectly help with representation (Croatian being the same* language, but it does not use cyrillic script)
* Contentious point maybe, but they are mutually intelligible with what would otherwise be considered only dialectical variations
[0] https://digital-strategy.ec.europa.eu/en/news/commission-wel...
but yeah, anything alphabetic yet not latin gets the worst tradeoff.
Not really, because I don't think anyone use the whole training dataset to train the tokenizer.
From my brief (disclosure: claude-assisted [0]) internet search I learned that typically they use from tens to hundreds GB of data to train the tokenizer. Sometimes it is hand-curated, sometimes it is randomly sampled.
Of course, you can deliberately choose how much text in each language do you include in the training set for the BPE and deliberately optimize for (or against) a specific language.
[0] https://claude.ai/share/da277189-f23a-4071-bd9f-2d75fe6aa779
I don't know whether European languages work inherently better with tokenizers, but I don't see why they would.
> The researchers found the energy and accuracy disparity by measuring the amount of energy used by three open-source AI models, Alibaba’s Qwen 3, Google’s Gemma 3, and Meta’s Llama 3.1.
Edit: sorry, i went straight to the paper. I agree, the article itself (and the headline) are bad (as they tend to be on this topic)
Working out the kinks in your methodology obviously takes some time. But is there any structural reason not to rerun the benchmarks on current models in the final stages before submission? Assuming the results generalize to those newer models
I am hopeful that we can eventually move to open weights models, but at the time we did the experiments, the open weights models did not have the capabilities we needed. The ever-shifting sands of frontier models makes reproducibility difficult, and not just for scientists.
Given the same model the only variable is the token count. The more tokens, the more passes through the LLM (to generate the next output token), the more energy used.
It's well known that non-English text consumes more tokens [1], [2] and thus more energy. This is due to two factors:
1. Latin script languages having different character pairings to English and modified characters, including language features like modifying verbs for person/tense/etc. (e.g. Spanish);
2. Non-Latin scripts that take more bytes to represent in UTF-8 and (esp. for ideographic and syllabic languages like Chinese, Japanese, and Korean) have lower common character pair sequences, meaning shorter tokens for those scripts and thus more tokens for a given text sequence.
[1] https://ikriv.com/blog/?p=5322 "LLM tokens and foreign languages" (5 Aug 2025)
[2] https://medium.com/@programmerraja/how-your-llm-costs-5x-mor...
It is not exactly accurate, because it ignores speculative decoding which can often save a lot of time, compute and energy.
See this branch: https://news.ycombinator.com/item?id=50441444#50497854
From the table on page 6 [1]
> Token explosion. The sentence repeated 45 times.
Figure 4/table 3 is pretty interesting. It shows that essentially all languages clustered at 0.1J/token, with a few outshots up to in some cases 4.8J/token.
They have 900 runs per language, so that's a pretty good sample size.
Accuracy uses the paper's strict "#### N" parser and lenient fallback. acc is strict/lenient percent. len_cap is the number of responses truncated at the generation-length limit, not an energy cap. Each language has 50 responses.
nw-flash (DeepSeek V4.1 Flash 1017)
Language J/req xEN tok/req J/tok acc len_cap
English 23.4 1.00 174 0.134 100/100 0
Chinese 16.8 0.72 389 0.043 96/96 0
Russian 18.8 0.80 352 0.054 96/96 0
French 25.1 1.07 272 0.092 98/98 0
S. Pashto 69.2 2.96 752 0.092 92/92 1
Tigrinya 65.3 2.79 2299 0.028 0/76 8
Shan 98.5 4.21 2823 0.035 48/54 14
Tibetan 42.3 1.81 2227 0.019 0/76 9
nw-small (Qwen3.8-27B-FP8)
Language J/req xEN tok/req J/tok acc len_cap
English 62.2 1.00 300 0.207 100/100 0
Chinese 154.1 2.48 605 0.255 94/94 1
Russian 147.7 2.37 576 0.257 96/96 1
French 163.3 2.62 323 0.506 98/98 0
S. Pashto 777.2 12.49 1510 0.515 84/84 6
Tigrinya 823.8 13.24 2458 0.335 0/76 10
Shan 1407.4 22.62 3422 0.411 48/52 20
Tibetan 1455.5 23.39 3199 0.455 0/56 20
nw-large (Kimi K3.1 Pro)
Language J/req xEN tok/req J/tok acc len_cap
English 169.5 1.00 391 0.434 100/100 0
Chinese 113.8 0.67 491 0.232 98/98 0
Russian 78.7 0.46 527 0.150 98/98 0
French 298.3 1.76 406 0.735 98/98 0
S. Pashto 119.7 0.71 903 0.133 96/96 0
Tigrinya 1096.6 6.47 1870 0.586 38/86 2
Shan 503.6 2.97 2895 0.174 52/52 12
Tibetan 593.2 3.50 2298 0.258 0/82 4
Endpoint Output tokens Energy/request J/output token
nw-flash 6.83x 3.27x 0.48x
nw-small 5.87x 8.46x 1.44x
nw-large 4.39x 3.50x 0.80x Language Accept Tokens/step 0/3 steps 3/3 steps Output tok/req
English 76.53% 3.296 9.4% 62.8% 549
Chinese 73.37% 3.201 11.3% 58.2% 754
Russian 73.88% 3.217 11.1% 58.5% 739
French 75.50% 3.265 9.8% 60.9% 631
S. Pashto 70.44% 3.113 12.9% 54.5% 1332
Tigrinya 79.72% 3.391 8.2% 67.8% 3643
Shan 75.64% 3.269 11.2% 63.0% 3675
Tibetan 74.67% 3.240 10.3% 60.2% 2857
Overall 75.75% 3.273 10.3% 62.2% 1772 Language Strict Lenient Cap Tok/req J/req J/tok Req x EN Tok x EN J/tok x EN
English 100% 100% 0 548.7 230.33 0.4198 1.00 1.00 1.00
Chinese 96% 96% 1 754.0 694.47 0.9210 3.02 1.37 2.19
Russian 96% 98% 2 738.8 592.15 0.8015 2.57 1.35 1.91
French 100% 100% 0 631.0 1101.84 1.7462 4.78 1.15 4.16
S. Pashto 88% 98% 6 1332.1 208.06 0.1562 0.90 2.43 0.37
Tigrinya 56% 62% 19 3642.9 1445.84 0.3969 6.28 6.64 0.95
Shan 44% 54% 27 3674.8 3791.59 1.0318 16.46 6.70 2.46
Tibetan 2% 70% 15 2856.9 5306.68 1.8575 23.04 5.21 4.42I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.
Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.
https://partyfly.github.io/tokenizer-fairness-2026/
Already been using English to conserve tokens, and then translate the final result to other languages if necessary. I think this was already obvious to most international LLM users proficient in multiple languages.
It would be interesting to see if it would make any difference if you posttrained with translations of the same texts, i.e. focusing on teaching the model to specifically translate concepts between languages.
But I would subtly add it's also about the training data, which is optimized towards English. Not intentionally, just incidentally. (And this is a really hot area of improvement with a long way to go.)
Which is largely why the tokenization part that you mentioned is what it is.
I think you only start getting seriously bad impacts in non-Romance languages where they have their own alphabet, or use the English alphabet phonetically causing letters to appear next to each other that never would in English. E.g. in English "qu" is a pretty obvious token, you rarely see a Q without a U after it. Happens all the time in anglicized Middle Eastern words, though. Quran is like a whole other thing, you'd almost always expect another vowel after qu, not a consonant.
I don't know how true that is, and I can't imagine knowing more than one language. I'm impressed by anyone who knows more than one....
LLMs do what they say on the tin: they model language. If you ask an LLM "find security vulnerabilities in my code" for hello_world.rs, the next likely tokens are "I found the following vulnerabilities", regardless of what logic or reason say.
It happens to be true that predicting next tokens on your own output mocks actual reasoning fairly well. But it still just a simulacrum rather than the real thing, and when held as an abstraction it leaks everywhere.
Understanding this is critical to avoiding the common failure case of scope creep and code slop.
Trying to pretend otherwise at this point doesn’t help us to utilize them or understand their limits. We’re better off accepting that intelligence isn’t unique to us as a species. Then we can see that modeling language as a tool for us as a vehicle for thinking and reasoning, which philosophers like Wittgenstein already landed upon before LLMs.
Genuinely curious, not sacastic question below.
Is this one of those posts like "eating more sugar causes bad health" type of post because most of info on internet is in English? Or a new insight people was not aware of?