In summary: By using a uniform singular value spectrum 📊 and task-specific subspaces 🎯, Iso-CTS achieves state-of-the-art performance across all settings!🔥

🔍 That’s why we propose replacing the least important components with task-specific vectors that are orthogonal to the common subspace.

This further enhances alignment 🎯, and the performance naturally improves! 📈

This simple modification boosts task arithmetic by 📈 10-15% across all model merging scenarios, achieving state-of-the-art results in most cases!🔥

However, we found that the bottom components contribute very little to the final performance… 📉⚠️

Based on this, we propose an isotropic merging framework that:
📊 Flattens the singular value spectrum of task matrices
🎯 Enhances alignment between tasks
⚖️ Reduces the perf gap
Surprisingly, the best performance is achieved when the singular value spectrum is uniform!🚀

We show that alignment between singular components of task-specific & merged matrices strongly correlates with performance gains over the pre-trained model! 📈

🔍 Tasks that are well-aligned get amplified 🔊, while less aligned ones become underrepresented and struggle. 😬📉

🚀 What happens when you modify the spectrum of singular values of the merged task vector? 🤔

Apparently, you achieve 🚨state-of-the-art🚨 model merging results! 🔥

✨ Introducing “No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces”