This work is a collaboration with a team of talented researchers at the AreaSciencePark of Trieste, Italy.
Special thanks to @alexpietroserra.bsky.social, Alessio Ansuini and @albecazzaniga.bsky.social !

If you are @neuripsconf.bsky.social don't miss our poster tomorrow, Dec 11, at 11am!!
🧵6/6

⚒️ We applied an advanced density-based clustering algorithm, showing its potential as an interpretability tool and in guiding novel strategies for the effective finetuning of LLMs.
🧵5/6

In fine-tuning, answer-focused modes rapidly emerge midway through the network, just after the intrinsic dimension peak.
Early layers remain largely unchanged.
🧵4/6

In few-shot learning, the prompt topic defines the modes of data distribution early in the network, and density modes are hierarchically organized based on the similarity of the subjects.
🧵3/6

🎯 Key results: few-shot learning and fine-tuning show two distinct processing phases inside LLMs.

These phases are separated by a peak of the data intrinsic dimension and a sharp decrease in the separation of the probability modes.

Paper: arxiv.org/abs/2409.03662
🧵2/6