Check it out:
📰 Paper: doi.org/10.48550/arX...
💾 Code + data: osf.io/frqbe/files
Let us know what you find!

There’s lots more work to be done here, including tinkering with prompts, model parameters, and extending to freely-available LLMs. In the meantime, we hope this is useful to folks and complements existing tools with something new: fast, scalable, and customizable VFF estimation.

📌 VFFs from Gahl et al. (2004)'s manually annotated (i.e. gold-standard) VFFs
📌 against preferences for competing frames (the dative alternation and NP/SC ambiguity) 🧵6/8

We benchmarked it thoroughly. The LLM consistently outperformed benepar & the Stanford Parser:
📌 300 human-annotated sentences (LLM accuracy = 79%, vs. 69% for benepar and 59% for Stanford) 🧵5/8

That’s particularly exciting because existing datasets don’t scale well. They’re hard to adapt to new verbs/contexts/languages according to experimental need. Our pipeline is simple, scalable, and adaptable. We release the full code + VFF norms for 476 English verbs. 🧵4/8

So we got creative and tried asking an LLM to parse a bunch of sentences. As it turns out, not only did this work, but the LLM outperformed both the Stanford Parser and the Berkeley Neural Parser (benepar), a state-of-the-art deep-learning parser trained on treebanks. 🧵3/8

We needed syntactic norms for an experiment -- specifically Verb Frame Frequencies (VFFs), or how often particular verbs appear in different syntactic frames (e.g., intransitive, prepositional object, etc.). Nothing in the literature quite fit. 🧵2/8

Thank you, Florence!!

P.S. Yes, we know, Frankenstein wasn't the monster's name. 🤣

Read more here:
doi.org/10.1038/s442...
Work with my PI Adeen Flinker and our clinical team. So many thanks to labmates and everyone else who helped along the way! 🧵✂️

More broadly, the field has largely assumed that the representations we study with single word production tasks are the same as those involved in sentences. By successfully using models trained on picture naming to decode words in sentences, we verify this 🔑 point. 🧵8/9

These findings show that word processing doesn't always look like it does in picture naming: it depends on task demands. This complexity may even help explain why languages globally prefer placing subjects before objects! 🧵7/9

We took a closer look at what was going on in prefrontal cortex. This revealed that these sustained representations traced back to different regions depending on a word's sentence position: when it was a subject, it was encoded in IFG, while MFG encoded objects. 🧵6/9

In passive sentences like "Frankenstein was hit by Dracula", we observed sustained neural activity encoding BOTH nouns simultaneously throughout the entire utterance. This was particularly true in prefrontal cortex. 🧵5/9

For straightforward active sentences ("Dracula hit Frankenstein"), the brain activated words sequentially, matching their spoken order. But things changed dramatically for more complex sentences... 🧵4/9

We trained machine learning classifiers to identify each word's specific neural pattern. 🔑We ONLY used data from picture naming (single word production) to train the models. We then used the models to predict what word patients were saying in real time as they said sentences.🧵3

We recorded brain activity directly from cortex in neurosurgical patients (ECoG) while they used 6 words in two tasks: picture naming ("Dracula") and scene description ("Dracula hit Frankenstein"). 🧵2/9

Wow, thanks Laurel! Honestly one of the best compliments I’ve ever gotten given the quality of the other talks!