📄 Paper: arxiv.org/abs/2507.07207
💻 Code: github.com/smonsays/sca...

But, not all training distributions enable compositional generalization -- even with scale.
Strategically choosing the training data matters a lot.

We prove that MLPs can implement a general class of compositional tasks ("hyperteachers") using only a linear number of neurons in the number of modules, beating the exponential!

It turns out that simply scaling multilayer perceptrons / transformers can lead to compositional generalization.

Most natural data has compositional structure. This leads to a combinatorial explosion that is impossible to fully cover in the training data.

It might be tempting to think that we need to equip neural network architectures with stronger symbolic priors to capture this compositionality, but do we?

Would love to be added as well :)