There’s more to Neural Nets than big fat LLMs!

We’ve built a NN-agent framework to simulate how people choose the best word in a given communication context (i.e. pragmatic naming behavior).

With @yuqing0304.bsky.social, @ecesuurker.bsky.social, Tessa Verhoef, @gboleda.bsky.social

We also collect human acceptability judgements & show that *overall* harder phenomena for LLMs are also harder for people, but there are some notable exceptions.

TurBLiMP expands the shortlist of existing language-specific BLiMPs with 2 important properties: high word order freedom & agglutination.

To study LLMs' robustness to these properties, we create experimental paradigms testing syntactic skills w/ different word orders & subordination strategies:

This is hard, slow-paced work going well beyond benchmark translation (let alone LLM-assisted benchmark generation!) It requires real *linguistic* expertise & long discussions on what makes a phenomenon representative of a language. Here's our proposal, inspired by EnglishBLiMP w/ major adaptations:

In NeLLCom, neural-net agents learn to speak a pre-defined language and then use it to communicate in pairs or in groups. Observing how the agents’ language properties evolve during communication gives us insights on how specific human language properties may have evolved.

Thanks to @haspelmath.bsky.social I just discovered this great collection of hypotheses extracted from evolutionary linguistics and typology papers, represented as a graph where linguistic properties are linked to others via different relations.

correlation-machine.com/CHIELD/varia...