Who's presenting on subjectivity in annotation (human label variation, learning from disagreement, perspectivism) at #ACL2025?

papers by e.g. @liweijiang.bsky.social @tiancheng.bsky.social @gabriellalapesa.bsky.social @romanklinger.de

keynote @verenarieser.bsky.social

link to full list below ⤵️

I will be at #acl2025 to present "Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals’ Subjective Text Perceptions" ✨

Huge thank you to my collaborators Jiaxin Pei @paul-rottger.bsky.social Philipp Cimiano @davidjurgens.bsky.social @dirkhovy.bsky.social 🍰

more below

Attributes help most for annotators with unique sociodemographic profiles. Apparently, LLMs learn to use unique combinations as a proxy ID! Learning from individual-level examples provides richer information than knowing sociodemographics.

Ok, but surely attributes are much more useful when transferring to annotators not seen in training? This setting is rarely tested in NLP, so we built a separate partitioning of DeMo for this evaluation. Turns out no model improves over the baseline!

Sociodemographic prompting and models fine-tuned with only the text content are our baselines. We compare against fine-tuning with annotator attributes or unique annotator identifiers (IDs). Trends are clear: attributes help a bit, but IDs are much more accurate!

Can LLMs learn to simulate individuals' judgments based on their demographics?

Not quite! In our new paper, we found that LLMs do not learn information about demographics, but instead learn individual annotators' patterns based on unique combinations of attributes!

🧵

🗻🥾☀️🌨️ ALPS25 Winter school, here we go! Looking forward to a week of talks, projects, posters, meeting many other researchers and, well, hiking!