For more details on IssueBench, check out our paper and dataset release. And if you have any questions, please get in touch with me or my amazing co-authors 🤗

Paper: arxiv.org/abs/2502.08395
Data: huggingface.co/datasets/Pau...

Even the issues on which models diverge in stance remain largely the same: Writing about Chinese political issues, Grok falls in with other Western-origin LLMs while DeepSeek’s bias better matches fellow Chinese LLM Qwen.

For this final version of our paper, we added results for Grok and DeepSeek alongside GPT, Llama, Qwen, and OLMo.

Surprisingly, despite being developed in quite different settings, all models are very similar in how they write about different political issues.

Quick recap of our setup:

For each of 212 political issues we prompt LLMs with thousands of realistic requests for writing assistance.

Then we classify each model response for which stance it expresses on the issue at hand.

Let me know if I missed anything in the timetables, and please say hi if you want to chat about sociotechnical alignment, safety, the societal impact of AI, or related topics :) Here is a link to the timetable sheet 👇 See you around!

docs.google.com/spreadsheets...

Finally, I will be with @carolin-holtermann.bsky.social and @a-lauscher.bsky.social to present our work on evaluating geotemporal reasoning ability in LLMs. This will be in the Wednesday 1100 poster session:

aclanthology.org/2025.acl-lon...

I will also be at @tiancheng.bsky.social's oral *today at 1430* in the SRW. Tiancheng will present a non-archival sneak peek of our work on benchmarking the ability of LLMs to simulate group-level human behaviours:

bsky.app/profile/tian...

Otherwise, you can find me in the audience of the great @manueltonneau.bsky.social oral *today at 1410*. Manuel will present our work on a first global representative dataset of hate speech on Twitter:

bsky.app/profile/manu...

Finally, there's a couple of papers on *LLM persuasion* on the schedule today. Particularly looking forward to Jillian Fisher's talk on biased LLMs influencing political decision-making!

*pluralism* in human values & preferences (e.g. with personalisation) will also just
grow more important for a global diversity of users.

@morlikow.bsky.social is presenting our poster today at 1100. Also hyped for @michaelryan207.bsky.social's work and @verenarieser.bsky.social's keynote!

Measuring *social and political biases* in LLMs is more important than ever, now that >500 million people use LLMs.

I am particularly excited to check out work on this by @kldivergence.bsky.social @1e0sun.bsky.social @jacyanthis.bsky.social @anjaliruban.bsky.social

For sure -- question format can definitely have some effect, and humans are also inconsistent. The effects we observed for LLMs in our paper though went well beyond what one could reasonably expect for humans. All just goes to show we need more realistic evals 🙏

I also find it striking that the article does not discuss at all in what ways / on which issues the models have supposedly become more "right-wing". All they show is GPT moves slightly towards the center of the political compass, but what does that actually mean? Sorry if I sound a bit frustrated 😅

Thanks, Marc! I would not read too much into these results tbh. The PCT has little to do with how people use LLMs, and the validity of the testing setup used here is very questionable. We actually had a paper on exactly this at ACL last year, if you're interested: aclanthology.org/2024.acl-lon...

Thanks, Marc. My intuition is that model developers may be more deliberate about how they want their models to behave than you frame it here (see GPT model spec or Claude constitution). So I think a lot of what we see is downstream from intentional design choices.

For claims about *political* bias we can then compare model issue bias to voter stances, as we do towards the end of the paper.

Thanks, Jacob. We also discussed this when writing the paper. In the end, our definition of issue bias (see 2nd tweet in the thread, or better the paper) is descriptive, not normative. At the issue level we say ”bias = clear stance tendency across responses“. Does that make sense to you?

We are very excited for people to use and expand IssueBench. All links are below. Please get in touch if you have any questions 🤗

Paper: arxiv.org/abs/2502.08395
Data: huggingface.co/datasets/Pau...
Code: github.com/paul-rottger...

It was great to build IssueBench with amazing co-authors @valentinhofmann.bsky.social Musashi Hinck @kobihackenburg.bsky.social @valentinapy.bsky.social Faeze Brahman and @dirkhovy.bsky.social .

Thanks also to the @milanlp.bsky.social RAs, and Intel Labs and Allen AI for compute.