Fernando Diaz
LightNews LightNews
banner
841io.bsky.social
Fernando Diaz
@841io.bsky.social
1.7K followers 620 following 90 posts
Associate Professor, CMU. Researcher, Google. Evaluation and design of information retrieval and recommendation systems, including their societal impacts.
Posts Replies Media Videos
Reposted by Fernando Diaz
arxiv-cs-ir.bsky.social
Rikiya Takehi, Fernando Diaz, Tetsuya Sakai
Diversification as Risk Minimization
https://arxiv.org/abs/2510.22681
October 28, 2025 at 5:52 AM
Reposted by Fernando Diaz
zouharvi.bsky.social
📢 Announcing the First Workshop on Multilingual and Multicultural Evaluation (MME) at #EACL2026 🇲🇦

MME focuses on resources, metrics & methodologies for evaluating multilingual systems! multilingual-multicultural-evaluation.github.io

📅 Workshop Mar 24–29, 2026
🗓️ Submit by Dec 19, 2025
October 20, 2025 at 10:37 AM
Reposted by Fernando Diaz
facct.bsky.social
We’re excited to release the Call for Papers for #FAccT2026 which will be held in Montreal, Canada in June 2026! Abstracts are due on January 8th, papers due on January 13th.

Call for Papers: facctconference.org/2026/cfp

Important info in thread →
ACM FAccT - 2026 CFP
facctconference.org
October 17, 2025 at 1:27 PM
Reposted by Fernando Diaz
yoshuabengio.bsky.social
AI is evolving too quickly for an annual report to suffice. To help policymakers keep pace, we're introducing the first Key Update to the International AI Safety Report. 🧵⬇️

(1/10)
October 15, 2025 at 10:49 AM
Reposted by Fernando Diaz
dmshanmugam.bsky.social
I am on the job market this year! My research advances methods for reliable machine learning from real-world data, with a focus on healthcare. Happy to chat if this is of interest to you or your department/team.
October 14, 2025 at 3:45 PM
Reposted by Fernando Diaz
cshc-bot.bsky.social
Renee Shelby, Fernando Diaz, Vinodkumar Prabhakaran: Taxonomy of User Needs and Actions https://arxiv.org/abs/2510.06124 https://arxiv.org/pdf/2510.06124 https://arxiv.org/html/2510.06124
October 8, 2025 at 6:32 AM
Reposted by Fernando Diaz
djfoster.bsky.social
MSR NYC is hiring spring and summer interns in AI/ML/RL!

Apply here: jobs.careers.microsoft.com/global/en/jo...
Microsoft Research Lab - New York City - Microsoft Research
Apply for a research position at Microsoft Research New York & collaborate with academia to advance economics research, prediction markets & ML.
www.microsoft.com
October 2, 2025 at 8:58 PM
841io.bsky.social
In January, Asia Biega (MPI), Georgina Born (UCL), Mary Gray (MSR), Rida Qadri (G), and I ran a Dagstuhl Seminar bringing together folks from CS and the broader social sciences to discuss questions around AI and culture. Dagstuhl has just posted our report, 1/3

drops.dagstuhl.de/storage/04da...
October 2, 2025 at 6:55 PM
Reposted by Fernando Diaz
aolteanu.bsky.social
This was accepted to #NeurIPS 🎉🎊

TL;DR Impoverished notions of rigor can have a formative impact on AI work. We argue for a broader conception of what rigorous work should entail & go beyond methodological issues to include epistemic, normative, conceptual, reporting & interpretative considerations
aolteanu.bsky.social Alexandra Olteanu @aolteanu.bsky.social · Jun 18
We have to talk about rigor in AI work and what it should entail. The reality is that impoverished notions of rigor do not only lead to some one-off undesirable outcomes but can have a deeply formative impact on the scientific integrity and quality of both AI research and practice 1/
Print screen of the first page of a paper pre-print titled "Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor" by Olteanu et al. Paper abstract: "In AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly applied. We argue that this narrow conception of rigor has contributed to the concerns raised by the responsible AI community, including overblown claims about AI capabilities. Our position is that a broader conception of what rigorous AI research and practice should entail is needed. We believe such a conception -- in addition to a more expansive understanding of (1) methodological rigor -- should include aspects related to (2) what background knowledge informs what to work on (epistemic rigor); (3) how disciplinary, community, or personal norms, standards, or beliefs influence the work (normative rigor); (4) how clearly articulated the theoretical constructs under use are (conceptual rigor); (5) what is reported and how (reporting rigor); and (6) how well-supported the inferences from existing evidence are (interpretative rigor). In doing so, we also aim to provide useful language and a framework for much-needed dialogue about the AI community's work by researchers, policymakers, journalists, and other stakeholders."
September 29, 2025 at 11:13 PM
841io.bsky.social
I'm noticing that some conversational AI interfaces are pausing text generation (or at least rendering) until the user scrolls. A thread on attention modeling. 👀 🧵 1/10
September 24, 2025 at 8:30 PM
Reposted by Fernando Diaz
mdudik.bsky.social
🚨Microsoft Research NYC is hiring🚨

We're hiring postdocs and senior researchers in AI/ML broadly, and in specific areas like test-time scaling and science of DL. Postdoc applications due Oct 22, 2025. Senior researcher applications considered on a rolling basis.

Links to apply: aka.ms/msrnyc-jobs
Microsoft Research Lab - New York City - Microsoft Research
Apply for a research position at Microsoft Research New York & collaborate with academia to advance economics research, prediction markets & ML.
aka.ms
September 18, 2025 at 2:37 PM
841io.bsky.social
Giuliano, V.E. and P. E. Jones, "Linear Associative Information Retrieval", Rept. no. CACL-2, Arthur D. Little, Inc., Cambridge, Mass., Nov 1962, 240 p. Also in P. W. Howerton and D. C . Weeks [eds ]. "Vistas in Information Handling", 1963, p. 30-54.
ACORN-i: (Associative Content Retrieval Network) Wired for 40 sentences using 40 index terms. To pose an inquiry, the wires with the clips are attached to the terminals for the index terms and/or sentences deemed to be relevant by the user. As the large knob is turned the voltages on these wires are raised, and the neon bulbs light up in the order of "relevance" of the various sentences. Rel ative voltages on the individual wires are controlled by the other knobs. Association may be set either "free" or "narrow" by varying the setting on the lower right hand knob.
September 4, 2025 at 1:30 PM
Reposted by Fernando Diaz
nickseaver.website
junior faculty in STSy positions, check out the CIFAR global scholars call: cifar.ca/next-generat... I'm on the "Future Flourishing Program," and it's a nice community (and a good bit of money as well).
CIFAR Azrieli Global Scholars Program - CIFAR
Support and training for the research leaders of tomorrow.
cifar.ca
September 3, 2025 at 8:56 PM
Reposted by Fernando Diaz
lukestark.bsky.social
My university has announced a fund to essentially poach doctoral students from US institutions. DM me if you do work on the history/social impacts of AI and are interested in being poached 😂
July 17, 2025 at 8:17 PM
Reposted by Fernando Diaz
maik-froebe.bsky.social
To Eun Kim just presented the work on "Tip of the Tongue Query Elicitation for Simulated Evaluation" at #SIGIR2025. The approach will be used in the #TREC2025 Tip-of-the-Tongue track, and we had some sweets at the poster :)

The paper is available online: dl.acm.org/doi/10.1145/...
July 15, 2025 at 2:30 PM
Reposted by Fernando Diaz
maik-froebe.bsky.social
Lukas Gienapp presents "The Viability of Crowdsourcing for RAG Evaluation" at #SIGIR2025

The paper is available at: webis.de/publications...
July 15, 2025 at 1:53 PM
Reposted by Fernando Diaz
bmitra.bsky.social
Hello TREC-ToTers!

We have released the test queries for the TREC 2025 Tip-of-the-Tongue (TREC-ToT) Track. Please see the guidelines for more information: trec-tot.github.io/guidelines. Run submission deadline will tentatively be in August. #TREC2025 #TRECToT #TREC2025ToT

Please spread the word!
July 13, 2025 at 4:47 PM
Reposted by Fernando Diaz
sashamtl.bsky.social
I've always felt that Montréal is a safe space for research -- it’s a city that values science, collaboration, and diversity, and I’m proud to call it home 🏠

Like I said in the video: Montréal is home—and all are welcome.💖🦋

youtu.be/dWiL4b7-KEA
R&D in Montréal | Montréal Loves Researchers | Harness the Power of your Research in Canada
YouTube video by Montréal International
youtu.be
July 8, 2025 at 12:36 PM
Reposted by Fernando Diaz
facct.bsky.social
We're happy to officially announce the location of #FAccT2026!

Next year's conference will be held in Montreal, Canada 🇨🇦

Su Lin Blodgett and Zeerak Talat will be General Chairs, and Michael Madaio will be PC Chair 🎉

(thanks to MindView for the photo!)
Photo of the Town Hall meeting at last week's conference. The audience watches as Su Lin Blodgett announces the location of next year's conference. Su Lin Blodgett and Zeerak Talat will be General Chairs, and Michael Madaio will be PC Chair.
June 30, 2025 at 11:12 AM
841io.bsky.social
p epic conclusion in the og description of the sign test.

John Arbuthnot. An argument for divine providence, taken from the constant regularity observ'd in the births of both sexes. Philosophical Transactions of the Royal Society of London, 27(328):186-190, 1710.
From hence it follows, that Polygamy is Contrary to the Law of Nature and Justice, and to the Propagation of Human Race; for where Males and and Females are in equal number, if one Male takes Twenty Wives, Nineteen Men must live in Celibacy, whith is repugnant to the Design of Nature; nor is it probable that Twenty Women will be so well impreg­nated by one Man as by Twenty.
June 29, 2025 at 9:28 PM
Reposted by Fernando Diaz
aolteanu.bsky.social
There is a lot of talk and effort to figure out how genAI is different (I am also guilty of this!) -- the reality is that genAI is not that different and genAI is not that new either; it was hard to evaluate in the past, and it is still as hard to evaluate now #facct2025
June 23, 2025 at 7:17 AM
Reposted by Fernando Diaz
aolteanu.bsky.social
We have to talk about rigor in AI work and what it should entail. The reality is that impoverished notions of rigor do not only lead to some one-off undesirable outcomes but can have a deeply formative impact on the scientific integrity and quality of both AI research and practice 1/
Print screen of the first page of a paper pre-print titled "Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor" by Olteanu et al. Paper abstract: "In AI research and practice, rigor remains largely understood in terms of methodological rigor -- such as whether mathematical, statistical, or computational methods are correctly applied. We argue that this narrow conception of rigor has contributed to the concerns raised by the responsible AI community, including overblown claims about AI capabilities. Our position is that a broader conception of what rigorous AI research and practice should entail is needed. We believe such a conception -- in addition to a more expansive understanding of (1) methodological rigor -- should include aspects related to (2) what background knowledge informs what to work on (epistemic rigor); (3) how disciplinary, community, or personal norms, standards, or beliefs influence the work (normative rigor); (4) how clearly articulated the theoretical constructs under use are (conceptual rigor); (5) what is reported and how (reporting rigor); and (6) how well-supported the inferences from existing evidence are (interpretative rigor). In doing so, we also aim to provide useful language and a framework for much-needed dialogue about the AI community's work by researchers, policymakers, journalists, and other stakeholders."
June 18, 2025 at 11:48 AM
Reposted by Fernando Diaz
lindiatjuatja.bsky.social
When it comes to text prediction, where does one LM outperform another? If you've ever worked on LM evals, you know this question is a lot more complex than it seems. In our new #acl2025 paper, we developed a method to find fine-grained differences between LMs:

🧵1/9
June 9, 2025 at 1:47 PM
Reposted by Fernando Diaz
shaily99.bsky.social
🖋️ Curious how writing differs across (research) cultures?
🚩 Tired of “cultural” evals that don't consult people?

We engaged with interdisciplinary researchers to identify & measure ✨cultural norms✨in scientific writing, and show that❗LLMs flatten them❗

📜 arxiv.org/abs/2506.00784

[1/11]
An overview of the work “Research Borderlands: Analysing Writing Across Research Cultures” by Shaily Bhatt, Tal August, and Maria Antoniak. The overview describes that We survey and interview interdisciplinary researchers (§3) to develop a framework of writing norms that vary across research cultures (§4) and operationalise them using computational metrics (§5). We then use this evaluation suite for two large-scale quantitative analyses: (a) surfacing variations in writing across 11 communities (§6); (b) evaluating the cultural competence of LLMs when adapting writing from one community to another (§7).
June 9, 2025 at 11:30 PM