✅ PLoP Consistently outperforms other strategies (Attn, MLP)
✅ Works across different post-training scenarios: supervised fine-tuning, reinforcement learning
✅ Minimal computational overhead
In the worst case, it ties with the best manual approach. Usually, it's better.

NFN measures the alignment between each module (in the pretrained model) and the finetuning task. NFN is a cheap metric that can be calculated in one forward pass. It is based on a large width analysis of module-data alignment and is well suited for LoRA finetuning.

LoRA is amazing for finetuning large models cheaply, but WHERE you place the adapters makes a huge difference. Most people are just guessing where to put them (Attention, MLP, etc).

Meet "PLoP" (Precise LoRA Placement) 🎯, our new method for automatic LoRA placement 🧵

Are we hitting a wall with AI scaling? 🤔

That "plateau" you're seeing in scaling law charts might not be a fundamental limit, but a sign of suboptimal scaling strategies! I wrote a blogpost about this:

www.soufianehayou.com/blog/plateau...