Thank you!!! Looking forward to seeing you at CHI!

😍 Please see details in the paper. It was so fun to work on this project with my amazing co-authors @hopeschroeder.bsky.social Kynnedy Simone Smith @s010n.bsky.social @ziangxiao.bsky.social @emtseng.bsky.social Danielle Bragg

📎Preprint: arxiv.org/abs/2501.12557
🐙Github: github.com/rrrrrrockpan...

‼️ Finally, given serious concerns around validity, reproducibility, risks, and consequences, we conclude with a practical list of 💡guiding questions💡 to prompt researchers’ reflection on the validity and appropriateness of their LLM-related work.

🌟 We found common consequences that authors have discussed, incl. economic harms, malicious use, etc. We hope this framework can help authors *anticipate* the consequences of their research more easily, something that I've been thinking about for a while: dl.acm.org/doi/10.1145/....

😮 A bit worrisome: of the 153 papers, 130 (84.98%) used or studied a variation of the closed GPT-family models; 59 conducted some form of systems or studies that prompted LLMs but did not release the prompts. This trend adds urgency to calls for examining reproducibility in HCI.

🎭 We identified five different roles that LLMs play: LLMs as system engines, LLMs as research tools, LLMs as participants & users, LLMs as objects of study, and Users’ Perceptions of LLMs. 🌟 We hope to help authors to identify how and when to use LLMs in their next project. 🌟

🌟 However, we found less diversity in the contribution types pursued. Sampled papers predominantly center on empirical (98.7%) and artifact (61.44%) contributions, but we see a need for theoretical and methodological contributions, as well as literature review and opinion pieces.

🌟 Our study shows that researchers study LLMs across a wide range of application domains, incl. Communication & Writing, Augmenting Capabilities, etc. We hope this can help researchers identify where their interests might fit, and how to develop these areas as LLMs proliferate.

🔎 We systematically reviewed 153 LLM-related papers from CHI 2020-24. We identified 10 diverse domains where LLMs have been applied, categorized 5 distinct roles in HCI research projects, and 22 key limitations and concerns raised by authors.