📈 Experiments show significant improvements: faster knowledge propagation compared to traditional, topology-agnostic approaches in various real-world network settings. (5/6)

🔗 This work introduces topology-aware knowledge propagation, which tailors how models and information are shared based on each device’s place in the network, leading to more effective learning overall. (3/6)

🤝 New advances in decentralized learning! "Topology-Aware Knowledge Propagation in Decentralized Learning" proposes a novel way to improve how information flows in distributed machine learning systems. Let’s break it down! 🧵 (1/6) (arxiv.org/abs/2505.11760)

6/ 🌍 Scalable Impact:
Our methods aren’t just for small models! We show that they scale effectively to larger LMs, providing robust memorization mitigation without compromising performance across different sizes of models. Exciting progress for real-world applications!

5/💡Best Approach:
Our proposed unlearning method, BalancedSubnet, outperforms others by effectively removing memorized info while maintaining high accuracy.

4/🧪 Key Findings:
Unlearning-based methods are faster and more effective than regularization or fine-tuning in mitigating memorization.

3/⚡Introducing TinyMem:
We created TinyMem, a suite of small, efficient models designed to help test and benchmark memorization mitigation techniques. TinyMem allows for quick experiments with lower computational costs.

2/ 🚨 Main Methods:
We test 17 methods—regularization, fine-tuning, and unlearning—5 of which we propose. These methods aim to remove memorized info from LMs while preserving performance.

1/🧵ICLR 2025 Spotlight Research on LM & Memorization!
Language models (LMs) often "memorize" data, leading to privacy risks. This paper explores ways to reduce that!
Paper: arxiv.org/pdf/2410.02159
Code: github.com/msakarvadia/...
Blog: mansisak.com/memorization/