It’s been a real pleasure visiting @devon-jarvis.bsky.social lab @caandl-lab Wits University this week and giving a talk – great discussions and inspiring work happening here! 🇿🇦

Had a wonderful first day @ CCN in Beautiful Amsterdam ! 🌷

🎓Thrilled to share I’ve officially defended my PhD!🥳

At @gatsbyucl.bsky.social, my research explored how prior knowledge shapes neural representations.

I’m deeply grateful to my mentors, @saxelab.bsky.social and @caswell.bsky.social, my incredible collaborators, and everyone who supported me!

Had so much fun presenting our two posters at ICLR with @devon-jarvis.bsky.social and Nico Anguita ! Thanks to everyone who came by and chatted!

3️⃣These findings have practical implications for:
Continual learning, Reversal learning, Transfer learning, Fine-tuning (7/9)

2️⃣ The transition from lazy to rich learning isn’t just about absolute scale—it depends on a complex interplay of architecture, relative scale, and weight magnitude. (6/9)

Key findings:
1️⃣ We derive exact solutions for gradient flow, representational similarity, and the finite-width Neural Tangent Kernel (NTK).
(5/9)

We focus on deep linear networks and introduce λ-balanced initialization, a framework that captures how the relative scale of weights across layers impacts the learning regime. (4/9)

Prior research identified two key learning regimes:
Lazy learning – Representations remain mostly static
Rich/feature learning – Representations evolve dynamically
Our work investigates how initialization determines where a network falls on this spectrum. (3/9)

🚀 An other Exciting news! Our paper "From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks" has been accepted at ICLR 2025!

arxiv.org/abs/2409.14623

A thread on how relative weight initialization shapes learning dynamics in deep networks. 🧵 (1/9)

⏩Continual learning implications:
We analyze Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) and show that specialization dynamics impact its effectiveness, revealing limitations in standard continual learning regularization.
(6/8)

Key findings:
1. In non-specialized networks, the relationship between task similarity and forgetting shifts—leading to a monotonic forgetting trend.
2. We identify initialization schemes that enhance specialization by increasing readout weight entropy and layer imbalance.
(5/8)

✍️Our approach:
1. Analyzing specialization dynamics in deep linear networks (Saxe et al., 2013).
2. Extending to high-dimensional mean-field networks trained with SGD (Saad & Solla, 1995; Biehl & Schwarze, 1995). (4/8)

💡 Key insight:
Specialization isn’t guaranteed—it strongly depends on initialization. We show that weight imbalance and high weight entropy encourage specialization, influencing feature reuse and forgetting in continual learning.
(3/8)

Our paper, “A Theory of Initialization’s Impact on Specialization,” has been accepted to ICLR 2025!
openreview.net/forum?id=RQz...
We shows how neural network can build specialized and shared representation depending on initialization, this has consequences in continual learning.
(1/8)

Had an amazing time with @marcofm.bsky.social @danielkunin.bsky.social @david-klindt.bsky.social and Feng Chen doing science 🧑‍🔬 and surfing 🌊 in Santa Barbara for the Follow-on Program at the Kavli Institute for Theoretical Physics! 🌴 #ScienceAndSurf #KITP