Running an ablation on a subset of miniF2F, we find that a model ablated to prevent the sharing of lemmas across tasks also exhibits strong performance.

Studying the LEGO-Prover (a system for formalizing natural language proofs by learning reusable lemmas), we find that lemma reuse is very uncommon, and no lemma reused twice.

Studying TroVE (a system that learns reusable python functions), we find only 3 instances of a learned function being reused correctly, out of 3,201 test questions in the MATH dataset. Furthermore, our libraryless ablation outperforms the original on 3 of 4 MATH splits tested.

LLM powered library learning systems achieve SoTA performance on several tasks, but is this driven by the reuse of learned tools? We study two library learning systems for mathematics and find that the reuse of learned tools is extremely infrequent and can harm performance 🧵

Same findings hold on a neurosymbolic NMN model, even though these models are specifically designed to be compositional!

We stratify value pairs (e.g., blue + sphere) by attribute diversity, i.e., the number of possible train-time alternative values for each attribute. Low diversity combinations have a larger systematicity gap (difference in accuracy between seen and unseen combinations)!

For 29 different pairs of held-out object attributes (e.g., rubber cylinders), we create separate train and test splits in a modified CLEVR setting. Combinations of certain values for this attribute pair will be present at test time, but not train.

Will multimodal models systematically generalize if trained on enough data? In a controlled VQA setting, we find it’s not data quantity, but data DIVERSITY that matters! 🧵

Joint w/ @ab-carrell.bsky.social @kumarkagrawal.bsky.social Yash Sharma @nsaphra.bsky.social
www.cs.toronto.edu/~ianberlot/d...