#ACL2025

🧑‍🤝‍🧑 @ecekt.bsky.social, @alberto-testoni.bsky.social
📍 Monday, July 28, 11:00-12:30, Hall 4/5

See you in Vienna! ✨ @aclmeeting.bsky.social

🧑‍🤝‍🧑 @michaelwhanna.bsky.social, @akoller.bsky.social, @andre-t-martins.bsky.social, @pmondorf.bsky.social, Vera Neplenbroek, Sandro Pezzelle, @barbaraplank.bsky.social, @davidschlangen.bsky.social, Alessandro Suglia, @akskuchi.bsky.social

2️⃣ LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (Main Conference)
🧑‍🤝‍🧑 @annabavaresco.bsky.social, @raffagbernardi.bsky.social, @leobertolazzi.bsky.social, @delliott.bsky.social, Raquel Fernández, Albert Gatt, @esamghaleb.bsky.social, Mario Giulianelli

[4/4] We hope to inspire future research into methods that counter the influence of stereotypical associations on the model’s latent representation of the user, particularly when the user’s demographic group is unknown.

Code and data:
github.com/Veranep/impl...

[3/4] Our findings reveal that LLMs infer demographic info based on stereotypical signals, sometimes even when the user explicitly identifies with a different demographic group. We mitigate this by intervening on the model’s internal representations using a trained linear probe.

[2/4] We systematically explore how LLMs respond to stereotypical cues using controlled synthetic conversations, by analyzing the models’ latent user representations through both model internals and generated answers to targeted user questions.

[4/4] We hope to inspire future research into methods that counter the influence of stereotypical associations on the model’s latent representation of the user, particularly when
the user’s demographic group is unknown.

Code and data: github.com/Veranep/impl...

[3/4] Our findings reveal that LLMs do infer demographic info based on stereotypical signals, sometimes even when the user explicitly identifies with a different demographic group. We mitigate this by intervening on the model’s
internal representations using a trained linear
probe.