Posted by syrusakbary 3 hours ago
Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.
The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.
Finally, interactions with Python are minimized, and threads have minimal interactions with each other.
Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!
Source: https://www.gartner.com/en/newsroom/press-releases/2026-07-2...
Latency can be just as important as overall throughput, especially for inference providers like Groq and Cerebras.
From what I can tell it's also useful for inference when considering time-to-first-token (TTFT) as reported by fastokens.[0]
I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache. If you have a long prefix that's been seen before (say a system prompt), the time for tokenizing that will be a large part of your TTFT. The tokenizer cache should be warmed up in this case, so the throughput for Gigatoken would be significantly higher than reported in the repo.
Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determined there is a hit. That means tokenisation can proceed in parallel with your cache query, and since these caches are distributed in production systems I imagine the query itself could be slow.
I'm not trying to undermine the utility, and this is obviously excellent work. Being able to tokenise faster on the client also seems useful (precise token counts for context pruning heuristics, instead of `chars / 4`), and on a phone your work translates directly to energy savings. I'm just curious about the cache lookup point.
In practice every training project I've worked on does tokenization in a separate data processing phase.
I have spent way too much time waiting 10-15 minutes tokenizing my training dataset only for the run to crash over some minor bug after that. (If I was smarter, I’d test on a smaller batch first.)
It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
A massive speedup like this is a nice efficiency savings on some of these data pipelines for sure.