⚙️ Data and Compute Efficiency of 𝗠𝗼𝗜𝗖𝗟: We find that 𝗠𝗼𝗜𝗖𝗟 is more efficient in terms of data and computation compared to conventional (concat-based) ICL! (🧵6/n)

📉 Noisy and Imbalanced Demonstrations: By assigning weights to each demonstration subset, 𝗠𝗼𝗜𝗖𝗟 can effectively handle various practical applications where data quality varies. (🧵5/n)

🌐Generalization to Unseen Demonstrations: 𝙨𝙘𝙖𝙡𝙖𝙧 weights require predefined demonstration subsets.
Using 𝙃𝙮𝙥𝙚𝙧-𝙣𝙚𝙩𝙬𝙤𝙧𝙠—a smaller fine-tuned hyper-network that dynamically generates weights for each expert based on all concatenated demonstration subsets. (🧵4/n)

📊 𝗠𝗼𝗜𝗖𝗟 in Classification Tasks: 𝗠𝗼𝗜𝗖𝗟 outperformed Baseline ICL on 5 out of 7 datasets!
Using 𝙨𝙘𝙖𝙡𝙖𝙧 weights—a vector of trainable parameters that assign each expert a weight—we fine-tuned how demonstration subsets are combined. (🧵3/n)

🚀 How does 𝗠𝗼𝗜𝗖𝗟 improve In-Context Learning? 𝗠𝗼𝗜𝗖𝗟 prompts an LLM with multiple demonstration subsets, obtaining multiple experts, and merges their predictions via a trainable weighting function—it doesn’t require any fine-tuning of the LLM parameters! (🧵2/n)

🤔How to achieve efficient ICL without storing a huge dataset in one prompt?
💡Mixtures of In-Context Learners (𝗠𝗼𝗜𝗖𝗟): we treat LLMs prompted with subsets of demonstrations as experts and learn a weighting function to optimise the distribution over the continuation (🧵1/n)