🚀Proud to share our work on the training dynamics in Transformers with Wassim Bouaziz & @viviencabannes.bsky.social @Inria @MetaAI

📝Easing Optimization Paths arxiv.org/pdf/2501.02362 (accepted @ICASSP 2025 🥳)

📝Clustering Heads 🔥https://arxiv.org/pdf/2410.24050

🖥️ github.com/facebookrese...

1/🧵

LLMs and today's AI research chase top-down intelligence through abstract reasoning. This path might get us to AGI faster but I don’t think it will help us understand how intelligence emerged in nature.

1/2

I am usually quite critical of LLM hypey breakthroughs, but I don’t see an issue with OpenAI training on ARC.
There is an issue, but it’s more fundamental and multifaceted.

1- As it’s often the case, the benchmark here has become the goal. @fchollet.bsky.social also said solving ARC != AGI

1/3

Kind of a broken record here but proceedings.neurips.cc/paper_files/...
is totally fascinating in that it postulates two underlying, measurable structures that you can use to assess if RL will be easy or hard in an environment

When I was 5, I loved science, which I took to be planets and magnets and chemicals and shit.

Soon I realized science was a collective human activity, and these were just the objects of its attention.

Anyway, whenever I read an OpEd on how science isn't political, I think "Are you 5 years old?"

Since discovering @drmichaellevin.bsky.social work, I have been fascinated by hierarchical multi-agent systems in which the hierarchy develops across different ontological levels.
So hierarchy that goes
engineers -> the company they work for,
rather than
engineers -> their boss.

If you're an RL researcher or RL adjacent, pipe up to make sure I've added you here!
go.bsky.app/3WPHcHg

Not sure another social media app will increase my productivity.

But at least I can justify this one as useful for work.