We’re still looking to expand the language coverage of Global PIQA, so sign up if you don’t see your language represented yet! bsky.app/profile/mrl-...

You can read the preprint here: arxiv.org/abs/2510.21909
We release the tokenizers on Hugging Face: huggingface.co/datasets/cat...

We found that one of the biggest predictors of token premium effects was whitespace usage. So we also trained SuperBPE tokenizers, which do not use whitespace pretokenizers. SuperBPE tokenizers demonstrate better compression and less extreme token premium effects.

While it’s possible to achieve the same compression for some sets of languages by manipulating vocabulary size, there are some languages which changing vocab size does not lead to the same compression.

We show that some languages need more vocabulary items to get the same compression. This suggests that multilingual tokenizers should allocate more or less vocab to different languages, which can help us design more equitable multilingual tokenizers.

We used the compression rates we got from our monolingual tokenizers to estimate the vocabulary size at which a tokenizer would reach a target compression rate. We used this to determine the “optimal” vocab size for each language. This significantly reduces token premium effects.

We trained 7000 monolingual tokenizers for 97 languages and a range of vocabulary sizes. There was no vocabulary size at which token premiums go away, though larger vocabularies unsurprisingly lead to better compression and slightly smaller token premiums.

Compression isn’t the only tokenizer metric that matters, but it directly determines how many computations a model needs to process text. That affects both training and inference cost. Ideally, we want compression rates to be as similar as possible across languages.

Yeah I have many thoughts about that post. I do have a follow up post brewing 👀 probably will be some months before I finish it though!

Yeah, I think the models do generally capture this well and with a lot of flexibility. I think when people have done morphological tokenization, it tends to be really rigid and fragile to anything OOD

I guess the idea is basically to map strings of text to some kind of abstract representation of meaning and grammar? Maybe the closest thing is morphological tokenization. But to do this fully you would kind of need to solve Language first

Thanks!

huggingface.co/blog/catheri...

huggingface.co/blog/catheri...