📄 arxiv.org/abs/2412.13169
🗓️ Monday, July 28, 14:00–15:30
📍Room 1.62, Session 3: IP-Orals

Looking forward to seeing you in Vienna #ACL2025! 👋
#LLMs #PublicOpinion #SurveyResearch

w/ Berk @carohaensch.bsky.social @xinpeng.bsky.social Markus @barbaraplank.bsky.social Frauke @assenmacher.bsky.social

doi.org/10.48550/arX... preprint also out now on arxiv since srm has a bit of a backlog

We are waiting to get the proofs but I will send you the manuscript via Email in the meantimd

Leah von der Heyde Bernd Weiss and myself have an accepted manuscript at SRM doing this comparison for several LLMs plus a finetuned model

Johanna Hölzl from Uni Mannheim is working a lot on the reliability of Google Trends; she would for sure answer questions via Email

Similar to other preprints released in the last few months, we want to highlight the challenges of using LLMs for synthetic data creation or even public opinion estomation. While LLMs like GPT-3 hold promise for data synthesis, researchers must currently be very cautious of biases in predictions!

GPT-3's predictions often didn't match actual survey data from the GLES, with a notable variance in accuracy across different political parties. It performed better in predicting voters of center- and left-leaning parties than right-leaning ones. #syntheticdata

Our approach: We created detailed personas reflecting the German voting population, including demographic and political-ideological characteristics. These personas were used to prompt GPT-3 for voting behavior predictions. This approach is similar to a paper by Argyle et al. with ANES data. #LLM

Challenges

More specific points from the feminist perspective

Important questions to ask

But as we say in germany, "da ist noch Luft nach oben" - we can do better

Some framework proposals already exist!

How can bias be mitigated when building systems?

Robots behaviour can serve as a validation for biased human behaviour!

Study asking: What is the average skin color for different professions in generative AI?

Will it all get worse with Generative AI? Probably yes

She first presents some examples from web search engines and other "old" systems