Luca Righetti
@lucarighetti.bsky.social
Research Open_Phil, co-host HearThisIdea. Views my own.
🔸10% Pledge at GivingWhatWeCan.
🔸10% Pledge at GivingWhatWeCan.
I swear every research org has struggled with: "How do we share more WIP without people treating it as final?"
Love how clicking @METR_Evals's new Notes page changes the whole site to handwritten font and chalk background.
Strong visual screaming "no seriously, this is rough".
Love how clicking @METR_Evals's new Notes page changes the whole site to handwritten font and chalk background.
Strong visual screaming "no seriously, this is rough".
October 18, 2025 at 1:01 AM
I swear every research org has struggled with: "How do we share more WIP without people treating it as final?"
Love how clicking @METR_Evals's new Notes page changes the whole site to handwritten font and chalk background.
Strong visual screaming "no seriously, this is rough".
Love how clicking @METR_Evals's new Notes page changes the whole site to handwritten font and chalk background.
Strong visual screaming "no seriously, this is rough".
Transparency in AI safety is critical for building trust and advancing our scientific understanding.
We hope STREAM will:
• Encourage more peer reviews of model cards using public info;
• Give companies a roadmap for following industry best practices.
We hope STREAM will:
• Encourage more peer reviews of model cards using public info;
• Give companies a roadmap for following industry best practices.
September 2, 2025 at 4:03 PM
Transparency in AI safety is critical for building trust and advancing our scientific understanding.
We hope STREAM will:
• Encourage more peer reviews of model cards using public info;
• Give companies a roadmap for following industry best practices.
We hope STREAM will:
• Encourage more peer reviews of model cards using public info;
• Give companies a roadmap for following industry best practices.
Our paper draws on interviews across governments, industry, and academia.
Together, these experts helped us narrow our key criteria to six categories,
all fitting on a single page.
(Any sensitive info can be shared privately with AISIs, so long as it's flagged as such)
Together, these experts helped us narrow our key criteria to six categories,
all fitting on a single page.
(Any sensitive info can be shared privately with AISIs, so long as it's flagged as such)
September 2, 2025 at 4:03 PM
Our paper draws on interviews across governments, industry, and academia.
Together, these experts helped us narrow our key criteria to six categories,
all fitting on a single page.
(Any sensitive info can be shared privately with AISIs, so long as it's flagged as such)
Together, these experts helped us narrow our key criteria to six categories,
all fitting on a single page.
(Any sensitive info can be shared privately with AISIs, so long as it's flagged as such)
How can we verify that AI ChemBio safety tests were properly run?
Today we're launching STREAM: a checklist for more transparent eval results.
I read a lot of model reports. Often they miss important details, like human baselines. STREAM helps make peer review more systematic.
Today we're launching STREAM: a checklist for more transparent eval results.
I read a lot of model reports. Often they miss important details, like human baselines. STREAM helps make peer review more systematic.
September 2, 2025 at 4:03 PM
How can we verify that AI ChemBio safety tests were properly run?
Today we're launching STREAM: a checklist for more transparent eval results.
I read a lot of model reports. Often they miss important details, like human baselines. STREAM helps make peer review more systematic.
Today we're launching STREAM: a checklist for more transparent eval results.
I read a lot of model reports. Often they miss important details, like human baselines. STREAM helps make peer review more systematic.
I've been procrastinating on this chart of all model card releases by OpenAI, GDM, and Anthropic:
• 4 cases of late safety results (out of 27, so ~15%)
• Notably 2 cases were late results showed increases in risk
• The most recent set of releases in August were all on time
x.com/HarryBooth5...
• 4 cases of late safety results (out of 27, so ~15%)
• Notably 2 cases were late results showed increases in risk
• The most recent set of releases in August were all on time
x.com/HarryBooth5...
August 29, 2025 at 5:55 PM
I've been procrastinating on this chart of all model card releases by OpenAI, GDM, and Anthropic:
• 4 cases of late safety results (out of 27, so ~15%)
• Notably 2 cases were late results showed increases in risk
• The most recent set of releases in August were all on time
x.com/HarryBooth5...
• 4 cases of late safety results (out of 27, so ~15%)
• Notably 2 cases were late results showed increases in risk
• The most recent set of releases in August were all on time
x.com/HarryBooth5...
Still, there's a clear gap between expert perceptions in biosecurity and actual AI progress.
Policy needs to stay informed. We need to update these surveys as we learn more, add more evals, and replicate predictions with NatSec experts.
Better evidence = better decisions
Policy needs to stay informed. We need to update these surveys as we learn more, add more evals, and replicate predictions with NatSec experts.
Better evidence = better decisions
July 1, 2025 at 3:09 PM
Still, there's a clear gap between expert perceptions in biosecurity and actual AI progress.
Policy needs to stay informed. We need to update these surveys as we learn more, add more evals, and replicate predictions with NatSec experts.
Better evidence = better decisions
Policy needs to stay informed. We need to update these surveys as we learn more, add more evals, and replicate predictions with NatSec experts.
Better evidence = better decisions
How much should we trust these results? All forecasts should be treated cautiously. But two things do help:
• Experts and superforecasters mostly agreed
• Those with *better* calibration predicted *higher* levels of risk
(That's not common for surveys of AI and extreme risk!)
• Experts and superforecasters mostly agreed
• Those with *better* calibration predicted *higher* levels of risk
(That's not common for surveys of AI and extreme risk!)
July 1, 2025 at 3:09 PM
How much should we trust these results? All forecasts should be treated cautiously. But two things do help:
• Experts and superforecasters mostly agreed
• Those with *better* calibration predicted *higher* levels of risk
(That's not common for surveys of AI and extreme risk!)
• Experts and superforecasters mostly agreed
• Those with *better* calibration predicted *higher* levels of risk
(That's not common for surveys of AI and extreme risk!)
The good news:
Experts said if AI unexpectedly increases biorisk, we can still control it – via AI safeguards and/or checking who purchases DNA.
(68% said they'd support one or both these policies; only 7% didn't.)
Action here seems critical for preserving AI's benefits.
Experts said if AI unexpectedly increases biorisk, we can still control it – via AI safeguards and/or checking who purchases DNA.
(68% said they'd support one or both these policies; only 7% didn't.)
Action here seems critical for preserving AI's benefits.
July 1, 2025 at 3:09 PM
The good news:
Experts said if AI unexpectedly increases biorisk, we can still control it – via AI safeguards and/or checking who purchases DNA.
(68% said they'd support one or both these policies; only 7% didn't.)
Action here seems critical for preserving AI's benefits.
Experts said if AI unexpectedly increases biorisk, we can still control it – via AI safeguards and/or checking who purchases DNA.
(68% said they'd support one or both these policies; only 7% didn't.)
Action here seems critical for preserving AI's benefits.
I think this is part of a larger trend.
LLMs have hit many bio benchmarks in the last year. Forecasters weren't alarmed by those.
But "AI matches a top team at virology troubleshooting" is different – it seems the first result that's hard to just ignore.
LLMs have hit many bio benchmarks in the last year. Forecasters weren't alarmed by those.
But "AI matches a top team at virology troubleshooting" is different – it seems the first result that's hard to just ignore.
July 1, 2025 at 3:09 PM
I think this is part of a larger trend.
LLMs have hit many bio benchmarks in the last year. Forecasters weren't alarmed by those.
But "AI matches a top team at virology troubleshooting" is different – it seems the first result that's hard to just ignore.
LLMs have hit many bio benchmarks in the last year. Forecasters weren't alarmed by those.
But "AI matches a top team at virology troubleshooting" is different – it seems the first result that's hard to just ignore.
How concerned should we be about AIxBio? We surveyed 46 bio experts and 22 superforecasters:
If LLMs do very well on a virology eval, human-caused epidemics could increase 2-5x.
Most thought this was >5yrs away. In fact, the threshold was hit just *months* after the survey. 🧵
If LLMs do very well on a virology eval, human-caused epidemics could increase 2-5x.
Most thought this was >5yrs away. In fact, the threshold was hit just *months* after the survey. 🧵
July 1, 2025 at 3:09 PM
How concerned should we be about AIxBio? We surveyed 46 bio experts and 22 superforecasters:
If LLMs do very well on a virology eval, human-caused epidemics could increase 2-5x.
Most thought this was >5yrs away. In fact, the threshold was hit just *months* after the survey. 🧵
If LLMs do very well on a virology eval, human-caused epidemics could increase 2-5x.
Most thought this was >5yrs away. In fact, the threshold was hit just *months* after the survey. 🧵
It's worth remembering, US bombings are lower than they used to be. I doubt AI has affected this trend – and it's too early to tell what will happen.
But we have now seen two actual cases this year (Palm Springs IVF + Las Vegas cyber-truck). This threat is no longer theoretical.
But we have now seen two actual cases this year (Palm Springs IVF + Las Vegas cyber-truck). This threat is no longer theoretical.
June 9, 2025 at 9:32 AM
It's worth remembering, US bombings are lower than they used to be. I doubt AI has affected this trend – and it's too early to tell what will happen.
But we have now seen two actual cases this year (Palm Springs IVF + Las Vegas cyber-truck). This threat is no longer theoretical.
But we have now seen two actual cases this year (Palm Springs IVF + Las Vegas cyber-truck). This threat is no longer theoretical.
And you can imagine scenarios far worse.
The suspect was an extreme pro-natalist (thinks life is wrong) and fascinated with nuclear.
His bomb didn't kill anyone (except himself), but his accomplice had a recipe similar to a larger explosive used in the OKC attack (killed 168).
The suspect was an extreme pro-natalist (thinks life is wrong) and fascinated with nuclear.
His bomb didn't kill anyone (except himself), but his accomplice had a recipe similar to a larger explosive used in the OKC attack (killed 168).
June 9, 2025 at 9:32 AM
And you can imagine scenarios far worse.
The suspect was an extreme pro-natalist (thinks life is wrong) and fascinated with nuclear.
His bomb didn't kill anyone (except himself), but his accomplice had a recipe similar to a larger explosive used in the OKC attack (killed 168).
The suspect was an extreme pro-natalist (thinks life is wrong) and fascinated with nuclear.
His bomb didn't kill anyone (except himself), but his accomplice had a recipe similar to a larger explosive used in the OKC attack (killed 168).
Notably, a counter-terror strategy is to have police spot suspicious activity in online forums, using that to start investigations and undercover stings.
If more terrorists shift to asking AIs instead of online, this will work less. Police should be aware of this blindspot.
If more terrorists shift to asking AIs instead of online, this will work less. Police should be aware of this blindspot.
June 9, 2025 at 9:32 AM
Notably, a counter-terror strategy is to have police spot suspicious activity in online forums, using that to start investigations and undercover stings.
If more terrorists shift to asking AIs instead of online, this will work less. Police should be aware of this blindspot.
If more terrorists shift to asking AIs instead of online, this will work less. Police should be aware of this blindspot.
By contrast, the suspect's (likely-but-unconfirmed) reddit account also tried asking questions but didn't get any helpful replies.
It's not hard to imagine why an AI that is always ready to answer niche queries and able to have prolonged back-and-forths would be a useful tool.
It's not hard to imagine why an AI that is always ready to answer niche queries and able to have prolonged back-and-forths would be a useful tool.
June 9, 2025 at 9:32 AM
By contrast, the suspect's (likely-but-unconfirmed) reddit account also tried asking questions but didn't get any helpful replies.
It's not hard to imagine why an AI that is always ready to answer niche queries and able to have prolonged back-and-forths would be a useful tool.
It's not hard to imagine why an AI that is always ready to answer niche queries and able to have prolonged back-and-forths would be a useful tool.
Still, AI *did* answer many questions about explosives.
The court documents disclose one example, which seems in-the-weeds about how to maximize blast damage.
Many AIs are trained not to help at this. So either these queries weren’t blocked or easy to bypass. That seems bad.
The court documents disclose one example, which seems in-the-weeds about how to maximize blast damage.
Many AIs are trained not to help at this. So either these queries weren’t blocked or easy to bypass. That seems bad.
June 9, 2025 at 9:32 AM
Still, AI *did* answer many questions about explosives.
The court documents disclose one example, which seems in-the-weeds about how to maximize blast damage.
Many AIs are trained not to help at this. So either these queries weren’t blocked or easy to bypass. That seems bad.
The court documents disclose one example, which seems in-the-weeds about how to maximize blast damage.
Many AIs are trained not to help at this. So either these queries weren’t blocked or easy to bypass. That seems bad.
It’s unclear how counterfactual the AI was.
A lot of info on bombs is already online and the suspect had been experimenting with explosives for years.
I'd guess it's unlikely AI made a big diff. for *this* suspect in *this* attack – but not to say it couldn't in other cases.
A lot of info on bombs is already online and the suspect had been experimenting with explosives for years.
I'd guess it's unlikely AI made a big diff. for *this* suspect in *this* attack – but not to say it couldn't in other cases.
June 9, 2025 at 9:32 AM
It’s unclear how counterfactual the AI was.
A lot of info on bombs is already online and the suspect had been experimenting with explosives for years.
I'd guess it's unlikely AI made a big diff. for *this* suspect in *this* attack – but not to say it couldn't in other cases.
A lot of info on bombs is already online and the suspect had been experimenting with explosives for years.
I'd guess it's unlikely AI made a big diff. for *this* suspect in *this* attack – but not to say it couldn't in other cases.
Three weeks ago a car bomb exploded outside an IVF clinic in California, injuring four people.
Now court documents against his accomplice show the terrorist asked AI to help build the bomb.
A thread on what I think those documents do and don't show 🧵…
x.com/CNBC/status...
Now court documents against his accomplice show the terrorist asked AI to help build the bomb.
A thread on what I think those documents do and don't show 🧵…
x.com/CNBC/status...
June 9, 2025 at 9:32 AM
Three weeks ago a car bomb exploded outside an IVF clinic in California, injuring four people.
Now court documents against his accomplice show the terrorist asked AI to help build the bomb.
A thread on what I think those documents do and don't show 🧵…
x.com/CNBC/status...
Now court documents against his accomplice show the terrorist asked AI to help build the bomb.
A thread on what I think those documents do and don't show 🧵…
x.com/CNBC/status...
OpenAI and Anthropic *both* warn there's a sig. chance that their next models might hit ChemBio risk thresholds -- and are investing in safeguards to prepare.
Kudos to OpenAI for consistently publishing these eval results, and great to see Anthropic now sharing a lot more too.
Kudos to OpenAI for consistently publishing these eval results, and great to see Anthropic now sharing a lot more too.
February 26, 2025 at 12:49 AM
OpenAI and Anthropic *both* warn there's a sig. chance that their next models might hit ChemBio risk thresholds -- and are investing in safeguards to prepare.
Kudos to OpenAI for consistently publishing these eval results, and great to see Anthropic now sharing a lot more too.
Kudos to OpenAI for consistently publishing these eval results, and great to see Anthropic now sharing a lot more too.
My GW estimate comes from eyeballing Sri Lanka's electricity generation on Feb 9th vs. the week before. You can see the coal plant shut down)
(h/t to @ElectricityMaps for collecting this data on almost every country in the world)
app.electricitymaps.com/zone/LK/72h...
(h/t to @ElectricityMaps for collecting this data on almost every country in the world)
app.electricitymaps.com/zone/LK/72h...
February 17, 2025 at 8:24 PM
My GW estimate comes from eyeballing Sri Lanka's electricity generation on Feb 9th vs. the week before. You can see the coal plant shut down)
(h/t to @ElectricityMaps for collecting this data on almost every country in the world)
app.electricitymaps.com/zone/LK/72h...
(h/t to @ElectricityMaps for collecting this data on almost every country in the world)
app.electricitymaps.com/zone/LK/72h...
Bizzare that a monkey can cause >10X the blackout damage of Russian hackers
February 17, 2025 at 8:24 PM
Bizzare that a monkey can cause >10X the blackout damage of Russian hackers
My verdict:
1 test suggests the "lower bound" lacks wet-lab skills; 4 can't rule it out. It's plausible o1 was ~fine to deploy, but it remains subjective.
The report is clearer and more nuanced, which helps build trust. The next one should go further—and include harder evals.
1 test suggests the "lower bound" lacks wet-lab skills; 4 can't rule it out. It's plausible o1 was ~fine to deploy, but it remains subjective.
The report is clearer and more nuanced, which helps build trust. The next one should go further—and include harder evals.
December 10, 2024 at 7:57 PM
My verdict:
1 test suggests the "lower bound" lacks wet-lab skills; 4 can't rule it out. It's plausible o1 was ~fine to deploy, but it remains subjective.
The report is clearer and more nuanced, which helps build trust. The next one should go further—and include harder evals.
1 test suggests the "lower bound" lacks wet-lab skills; 4 can't rule it out. It's plausible o1 was ~fine to deploy, but it remains subjective.
The report is clearer and more nuanced, which helps build trust. The next one should go further—and include harder evals.
Big picture:
AIs keep saturating dangerous capability tests. With o1 we “ratcheted up” from multiple-choice to open-ended evals. But that won’t hold for long.
We need harder evals—ones where if an AI succeeds that suggests a real risk. (No updates yet on OAI’s wet-lab study).
AIs keep saturating dangerous capability tests. With o1 we “ratcheted up” from multiple-choice to open-ended evals. But that won’t hold for long.
We need harder evals—ones where if an AI succeeds that suggests a real risk. (No updates yet on OAI’s wet-lab study).
December 10, 2024 at 7:57 PM
Big picture:
AIs keep saturating dangerous capability tests. With o1 we “ratcheted up” from multiple-choice to open-ended evals. But that won’t hold for long.
We need harder evals—ones where if an AI succeeds that suggests a real risk. (No updates yet on OAI’s wet-lab study).
AIs keep saturating dangerous capability tests. With o1 we “ratcheted up” from multiple-choice to open-ended evals. But that won’t hold for long.
We need harder evals—ones where if an AI succeeds that suggests a real risk. (No updates yet on OAI’s wet-lab study).
Some critical points:
Previously, I flagged o1-previews’ 69% score on the Gryphon eval might match PhDs.
Turns out, experts score 57%—so o1 passed this eval *months* ago. I hope OAI declares such results in future.
(I'd keep an eye on the multimodal eval with no PhD score yet)
Previously, I flagged o1-previews’ 69% score on the Gryphon eval might match PhDs.
Turns out, experts score 57%—so o1 passed this eval *months* ago. I hope OAI declares such results in future.
(I'd keep an eye on the multimodal eval with no PhD score yet)
December 10, 2024 at 7:57 PM
Some critical points:
Previously, I flagged o1-previews’ 69% score on the Gryphon eval might match PhDs.
Turns out, experts score 57%—so o1 passed this eval *months* ago. I hope OAI declares such results in future.
(I'd keep an eye on the multimodal eval with no PhD score yet)
Previously, I flagged o1-previews’ 69% score on the Gryphon eval might match PhDs.
Turns out, experts score 57%—so o1 passed this eval *months* ago. I hope OAI declares such results in future.
(I'd keep an eye on the multimodal eval with no PhD score yet)
Credit where it’s due. The new system card improved on the old one:
• More comparisons to PhD baselines (now exist for 3/5 evals vs. 0/3 before)
• Multiple-choice tests converted to open-ended, making them more realistic
• Clear acknowledgment these results are "lower bounds"
• More comparisons to PhD baselines (now exist for 3/5 evals vs. 0/3 before)
• Multiple-choice tests converted to open-ended, making them more realistic
• Clear acknowledgment these results are "lower bounds"
December 10, 2024 at 7:57 PM
Credit where it’s due. The new system card improved on the old one:
• More comparisons to PhD baselines (now exist for 3/5 evals vs. 0/3 before)
• Multiple-choice tests converted to open-ended, making them more realistic
• Clear acknowledgment these results are "lower bounds"
• More comparisons to PhD baselines (now exist for 3/5 evals vs. 0/3 before)
• Multiple-choice tests converted to open-ended, making them more realistic
• Clear acknowledgment these results are "lower bounds"
A few weeks ago, I “peer-reviewed” o1-preview's ChemBio safety card and highlighted some issues about its methodology.
Now that o1 is out, how does it stack up?
Better! (Though there’s still room for improvement.)
Here’s my new o1 scorecard. 🧵👇
Now that o1 is out, how does it stack up?
Better! (Though there’s still room for improvement.)
Here’s my new o1 scorecard. 🧵👇
December 10, 2024 at 7:57 PM
A few weeks ago, I “peer-reviewed” o1-preview's ChemBio safety card and highlighted some issues about its methodology.
Now that o1 is out, how does it stack up?
Better! (Though there’s still room for improvement.)
Here’s my new o1 scorecard. 🧵👇
Now that o1 is out, how does it stack up?
Better! (Though there’s still room for improvement.)
Here’s my new o1 scorecard. 🧵👇