Read details in an excellent thread by Jan Ludziejewski x.com/jahulas/stat...

🤯 This is still very counterintuitive to me—MoE models are more memory-efficient 💾 than dense models!

We verify this empirically, and along the way, we develop joint scaling laws for both MoE and dense models.

Są na świecie politycy, którzy wspierają innowacje ...

(do rozwagi dla naszych w PL)

Deadline, sweet deadline

I am totally in the FOMO mode, seeing #NeurIPS24 messages. I’m not there :( this year.

However, I recommend chatting with my co-authors and students :) @bartoszpiotrowski.bsky.social @albertqjiang.bsky.social @michalnauman @mateuszostaszewski.bsky.social

2 papers on the main track and spotlight

<NeurIPS spotlight> Scaling has done wonders for deep learning, but for a long time it failed in on-policy RL... until now! We show that when done appropriately, scaling leads to state-of-the-art results in a variety of continuous control tasks🔥

BRO: Bigger, Regularized, Optimistic! 🧵

My first post here :)

Excellent news from NeurIPS. Two papers in, including a spotlight.

1. Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe
2. Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control