LightNews LightNews
banner
sifal.bsky.social
@sifal.bsky.social
14 followers 170 following 0 posts
Posts Replies Media Videos
Reposted
nsaphra.bsky.social
Ever looked at LLM skill emergence and thought 70B parameters was a magic number? Our new paper shows sudden breakthroughs are samples from bimodal performance distributions across seeds. Observed accuracy jumps abruptly while the underlying accuracy DISTRIBUTION changes slowly!
Distributional Scaling Laws for Emergent Capabilities Rosie Zhao, Tian Qin, David Alvarez-Melis, Sham Kakade, Naomi Saphra In this paper, we explore the nature of sudden breakthroughs in language model performance at scale, which stands in contrast to smooth improvements governed by scaling laws. While advocates of "emergence" view abrupt performance gains as capabilities unlocking at specific scales, others have suggested that they are produced by thresholding effects and alleviated by continuous metrics. We propose that breakthroughs are instead driven by continuous changes in the probability distribution of training outcomes, particularly when performance is bimodally distributed across random seeds. In synthetic length generalization tasks, we show that different random seeds can produce either highly linear or emergent scaling trends. We reveal that sharp breakthroughs in metrics are produced by underlying continuous changes in their distribution across seeds. Furthermore, we provide a case study of inverse scaling and show that even as the probability of a successful run declines, the average performance of a successful run continues to increase monotonically. We validate our distributional scaling framework on realistic settings by measuring MMLU performance in LLM populations. These insights emphasize the role of random variation in the effect of scale on LLM capabilities.
February 25, 2025 at 10:33 PM
Reposted
natolambert.bsky.social
OpenAI releasing a model spec is one of my favorite things about them. All frontier labs should be doing this. Closes the gap between intentions and results, so you can interpret them better.

openai.com/index/sharin...
Sharing the latest Model Spec
We’ve made updates to the Model Spec based on external feedback and our continued research in shaping desired model behavior.
openai.com
February 12, 2025 at 10:39 PM
Reposted
anton-l.bsky.social
Introducing 📐FineMath: the best open math pre-training dataset with 50B+ tokens!

Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.

🤗 huggingface.co/datasets/Hug...

Here’s a breakdown 🧵
A plot showing increased performance of Llama-3.2-3B when pretrained on FineMath
December 19, 2024 at 3:55 PM