LoRA makes "intruder dimensions" in singular vectors that differ from the pre-trained model while full fine-tune remains spectrally ~ to the base model. The takeaway LoRA excels at instruction fine-tuning with smaller datasets, while full fine-tuning shines in continued pre-training scenarios.

github.com/HKUDS/LightRAG

Kimi K2 - ~5 million