If you’ve made it to the end of this thread, thanks for reading! I hope we can connect soon and chat about the role of all these works in making AI evaluation more valid, reliable, and actionable for the real-world! 🥳

I will present this tomorrow at 11:45am in Paper Session 6: Integrating AI into the Workplace. The preprint has more analyses of AI literacy's impact and discussion of the implications of this work for human autonomy in high-risk domains.
www.arxiv.org/abs/2509.04404

But a positive note is that when people took an implicit association test (commonly used for anti-bias training) before doing the resume-screening task, they increased their selection of stereotype-incongruent candidates by 12.7% regardless of how biased the AI model they interacted with was.

We showed people AI recommendations that had varying levels of racial bias and found that (at most) human oversight decreased bias in final outcomes by at most 15.2% which is still far from the outcome bias rates when no AI or unbiased AI was used.

3️⃣ Last but not least, I’ll be talking about a human-subjects experiment follow-up to my AIES 2024 paper with @aylincaliskan.bsky.social, also co-authored by Mattea Sim and Anna-Maria Gueorguieva!
arxiv.org/abs/2407.20371

I'll be discussion this work tomorrow at 3:15pm during Poster Session 4! Our preprint has additional findings and analyses comparing generated images to human baselines and measuring dimensions of skin tone other than lightness/darkness.

arxiv.org/abs/2508.17465

We also found that depictions of racial identities are getting more homogenized with successive releases of SD, reinforcing harmful ideas about what people with stigmatized identities "should" look like.

We found that the newest model (SD XL) tends to generate images with darker skin tones compared to SD v1.5 and v2.1, but it still over-represents dark skin tones for stigmatized identities compared to non-stigmatized identities.

2️⃣ In another work with Sourojit Ghosh and @aylincaliskan.bsky.social, we analyzed skin tones in images of 93 stigmatized identities using three versions of Stable Diffusion (SD).

Chat with me about this work at Poster Session 2 (today at 6:15pm) or read our preprint in full!

arxiv.org/abs/2508.11067

We also find that 89.4% of papers don’t provide detailed information about real-world implementation of their findings. Based on this, we made a Fact Sheet that to guide researchers in communicating findings in ways that enable model developers or downstream users to implement them appropriately.

1️⃣ Sourojit Ghosh and I conducted a review of AI bias literature and found that 82% do not provide an explicit definition of bias and 79.9% do not explore bias outside of binary gender bias. This means that only a particular subset of marginalized groups may benefit from AI bias research.

So: In 2017, Congress made this happen:

NIH: Proposed 22% cut --> 9% increase
NSF: Proposed 11% cut --> 4% increase
NOAA: Proposed 16% cut --> 4% increase

Obviously, 2025 is not 2017. A lot is diff now. But still:

💰 Congress, not WH, sets budgets.

📞 Public support & calls to Congress matter.

🤩 If you made it this far, thanks for reading! Be sure to also check out our research paper which quantified the risks these systems pose to different gender, race, and intersectional groups! (6/6)

arxiv.org/abs/2407.203...