[2/2]
This is part of Kaggle benchmarking initiative, spearingheading the effort for trustworthy and rigorous LLM evaluation!
www.kaggle.com/blog/announc...

[5/5] Paper: arxiv.org/abs/2504.10356
Dataset, code, leaderboard: github.com/facebookrese...

[4/5] We release a dev partition which you can download and use. We also have a secret test partition, which will not be released at this point in time to avoid accidental contamination. If you upload your models on huggingface, we can evaluate them for you and put you on our leaderboard!

[3/5] Our dataset shows that in general, LLMs answer questions better if they are asked in the language corresponding to the culture where the knowledge originated, showing that there is a lot of growth potential when it comes to cross lingual knowledge transfer.

[2/5] Bulgarians are much more likely to ask an LLM about Tsar Simeon, rather than queen Elizabeth.

We also provide translations to and from English of our dataset so that we can measure crosslingual knowledge transfer.

Absolutely. I am just looking for a way to make scaling laws map to actual task metrics. At the moment we have the situation of "look at how nice the nll goes down on task X when we scale the model", but no idea what the actual downstream task metric will be or when it'd saturate. Maybe impossible

We should get rid of those auxiliary tasks it shoehorns everybody into nll for scaling laws...

Pretty cool work! Did you also see neurips.cc/virtual/2023... ?

It would be nice if we can also make a figure out more linear metrics for evaluating popular LLM tasks...