Very interesting! Thanks for sharing. I'd love to pick up on this thread when our paths cross (hopefully sooner than later!)

Thanks Laura! Very interesting comment. I agree that variance is an important and underrepresented angle in this discourse. What are some of the valuable questions and/or insights that you think can from shifting focus toward variance?

The tldr of our study is captured in this short 3 minute video produced by Emma Richard:

www.youtube.com/watch?v=4Vdw...

An interesting direction for future research will be to explore whether (and ideally how) incorporating multimodal representations into LLMs shapes their capacity to emulate human embodied metaphorical reasoning, especially about metaphors relating to vision.

This work also connects to our prior work demonstrating systematic differences in color perception btwn vision classifiers (based on deep neural networks & transformers) and humans. www.sciencedirect.com/science/arti...

This work nicely complements a paper published last week showing that text-based LLMs struggle to recover sensorimotor aspects of human concepts: www.nature.com/articles/s41...

Huge kudos to Ethan Nadler for leading this effort, and for our fantastic global team of collaborators: Sofronia Ringold, Tom Williamson, Antoine Pepin, Iulia-Maria Comşa, Karim Jerbi, Srini Narayanan, and Lisa Aziz-Zadeh.

This was a massive interdisciplinary effort – including physicists, neuroscientists, social scientists, and AI researchers from @Google DeepMind.

This has implications for the ongoing conversation around whether and to what extent LLMs can be treated as meaningful emulators of human cognition for psychological and/or sociological studies.

This suggests perceptual experience plays a role in metaphorical reasoning. It further suggests that LLMs are limited in their ability to recover the embodied aspects of metaphorical reasoning from statistical correlations among words alone.

We show that LLMs’ struggle to reason coherently about novel color metaphors and are less likely to reference embodied experience in their interpretations. By contrast, painters exhibit the highest rate of embodied reasoning when interpreting novel color metaphors.

Thank you for your interest! We welcome any and all feedback.