Noah Haber
@whaleactually.com
econ, epi, stats, meta, causal inference mutant scientist, epistemic humility fairy godmother, chaos muppet.
doing researchy metasciencey stuff at the Center for Open Science
doing researchy metasciencey stuff at the Center for Open Science
"Elapsed time" is to Intention to Treat as "Moving time" is to Per-Protocol.
I will not explain any further.
I will not explain any further.
September 2, 2025 at 6:32 PM
"Elapsed time" is to Intention to Treat as "Moving time" is to Per-Protocol.
I will not explain any further.
I will not explain any further.
A few of the first recruitment attempts to US-based journals have been met with a "times are too uncertain to take on a project like this," as expected
Small potatoes in the scheme of things, but just another example of the astounding destruction of scientific progress happening right now.
Small potatoes in the scheme of things, but just another example of the astounding destruction of scientific progress happening right now.
Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
July 31, 2025 at 2:48 PM
A few of the first recruitment attempts to US-based journals have been met with a "times are too uncertain to take on a project like this," as expected
Small potatoes in the scheme of things, but just another example of the astounding destruction of scientific progress happening right now.
Small potatoes in the scheme of things, but just another example of the astounding destruction of scientific progress happening right now.
The "meta trial" idea is maybe the most ambitious thing I've ever gotten to try in real life.
In theory, it goes a LONG way towards solving the feasibility, logistics, and incentives problems inherent in multi-unit policy experiments at scale.
Not just for journal policy. For policy period.
In theory, it goes a LONG way towards solving the feasibility, logistics, and incentives problems inherent in multi-unit policy experiments at scale.
Not just for journal policy. For policy period.
What we would really want in the end is a BUNCH of compatible, but realistic and pragmatic trials of custom variations on this policy and get to wrap it all up in a nice meta analysis.
For the last year or so, we've been piloting a pretty wild approach to getting exactly that:
A study-in-a-kit
For the last year or so, we've been piloting a pretty wild approach to getting exactly that:
A study-in-a-kit
July 29, 2025 at 5:12 PM
The "meta trial" idea is maybe the most ambitious thing I've ever gotten to try in real life.
In theory, it goes a LONG way towards solving the feasibility, logistics, and incentives problems inherent in multi-unit policy experiments at scale.
Not just for journal policy. For policy period.
In theory, it goes a LONG way towards solving the feasibility, logistics, and incentives problems inherent in multi-unit policy experiments at scale.
Not just for journal policy. For policy period.
Reposted by Noah Haber
This meta trial idea is pretty cool. Eventually you could arrive at a situation where journals help researchers get individual credit for research labor on large collaborative projects
Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
July 29, 2025 at 3:22 PM
This meta trial idea is pretty cool. Eventually you could arrive at a situation where journals help researchers get individual credit for research labor on large collaborative projects
Reposted by Noah Haber
Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
July 28, 2025 at 7:21 PM
Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
July 28, 2025 at 7:21 PM
Hi folks! Want to reintroduce a thing I'm leading at @cos.io: the Registered Revisions (meta) Trial.
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
This project about new peer review policy and a WILD new way of doing actionable evidence generation via RCTs. A LOT of RCTs.
Now piloted and ready for the main stage, and looking for partners! 🧵👇
death to "is it A or is it B that causes Y"
long live "how much do A and B contribute to Y"
long live "how much do A and B contribute to Y"
July 10, 2025 at 6:33 PM
death to "is it A or is it B that causes Y"
long live "how much do A and B contribute to Y"
long live "how much do A and B contribute to Y"
Triangle folks: I've got water, a bunch of 5 gallon jugs, tools galore, and plenty of time to deliver food/water/whatever to where it needs to go.
Throw a DM my way if you need anything at all.
Throw a DM my way if you need anything at all.
July 7, 2025 at 4:50 PM
Triangle folks: I've got water, a bunch of 5 gallon jugs, tools galore, and plenty of time to deliver food/water/whatever to where it needs to go.
Throw a DM my way if you need anything at all.
Throw a DM my way if you need anything at all.
Welp, it finally happened: I messed up a thing in a study involving participants, caught it, and am working on submitted a complete report and mitigation plan of the issue to the IRB.
Fortunately the damage to my pride is almost certainly larger than any risk to participants.
But still. Humbling.
Fortunately the damage to my pride is almost certainly larger than any risk to participants.
But still. Humbling.
July 2, 2025 at 5:14 PM
Welp, it finally happened: I messed up a thing in a study involving participants, caught it, and am working on submitted a complete report and mitigation plan of the issue to the IRB.
Fortunately the damage to my pride is almost certainly larger than any risk to participants.
But still. Humbling.
Fortunately the damage to my pride is almost certainly larger than any risk to participants.
But still. Humbling.
o hai, i am in copenhagen, sciencing yur sciences
Gonna be hanging out at the International Conference on the Science of Science (ICSSI) in Copenhagen next week, chatting about weird meta experiments, repping @cos.io, and generally trying to be useful in all the chaos.
If you're around, come hang out!
If you're around, come hang out!
June 16, 2025 at 8:01 AM
o hai, i am in copenhagen, sciencing yur sciences
Gonna be hanging out at the International Conference on the Science of Science (ICSSI) in Copenhagen next week, chatting about weird meta experiments, repping @cos.io, and generally trying to be useful in all the chaos.
If you're around, come hang out!
If you're around, come hang out!
June 11, 2025 at 6:54 PM
Gonna be hanging out at the International Conference on the Science of Science (ICSSI) in Copenhagen next week, chatting about weird meta experiments, repping @cos.io, and generally trying to be useful in all the chaos.
If you're around, come hang out!
If you're around, come hang out!
Reposted by Noah Haber
RCT-only plus liberal use of guidelines for power, measurement error, heterogeneous treatment effects, examining attrition, etc. We like to set up methods as the gold standard but then put little back doors in rather than own up to our true uncertainty.
June 3, 2025 at 1:44 PM
RCT-only plus liberal use of guidelines for power, measurement error, heterogeneous treatment effects, examining attrition, etc. We like to set up methods as the gold standard but then put little back doors in rather than own up to our true uncertainty.
Remember folks: the proverbial "gold standard" for evidence was driven by our own mainstream institutional evidence based medicine movement.
A simple (read: stupid) set of rules dictating what's good (double blind RCT, meta-analysis, etc) and not (anything else).
What could possibly go wrong?
A simple (read: stupid) set of rules dictating what's good (double blind RCT, meta-analysis, etc) and not (anything else).
What could possibly go wrong?
June 3, 2025 at 1:41 PM
Remember folks: the proverbial "gold standard" for evidence was driven by our own mainstream institutional evidence based medicine movement.
A simple (read: stupid) set of rules dictating what's good (double blind RCT, meta-analysis, etc) and not (anything else).
What could possibly go wrong?
A simple (read: stupid) set of rules dictating what's good (double blind RCT, meta-analysis, etc) and not (anything else).
What could possibly go wrong?
Added my name.
There is a lot to be said about how the aesthetics of science and scientific critique are beiung twisted, coopted, corrupted, and targeted for destructive ends.
Right now, it is time to make yourself heard, however you choose to do it.
actionnetwork.org/petitions/op...
There is a lot to be said about how the aesthetics of science and scientific critique are beiung twisted, coopted, corrupted, and targeted for destructive ends.
Right now, it is time to make yourself heard, however you choose to do it.
actionnetwork.org/petitions/op...
May 27, 2025 at 5:22 PM
Added my name.
There is a lot to be said about how the aesthetics of science and scientific critique are beiung twisted, coopted, corrupted, and targeted for destructive ends.
Right now, it is time to make yourself heard, however you choose to do it.
actionnetwork.org/petitions/op...
There is a lot to be said about how the aesthetics of science and scientific critique are beiung twisted, coopted, corrupted, and targeted for destructive ends.
Right now, it is time to make yourself heard, however you choose to do it.
actionnetwork.org/petitions/op...
Coming soon: the most fully realized preregistration/registered report ever (I think)
* Pre-written methods and results section
* Full reproducible data -> manuscript text and fig pipeline
* Pre-populated data w/ full trial(s) simulation
* Multiple levels (trials + meta analysis) for all
* Pre-written methods and results section
* Full reproducible data -> manuscript text and fig pipeline
* Pre-populated data w/ full trial(s) simulation
* Multiple levels (trials + meta analysis) for all
May 13, 2025 at 6:14 PM
Coming soon: the most fully realized preregistration/registered report ever (I think)
* Pre-written methods and results section
* Full reproducible data -> manuscript text and fig pipeline
* Pre-populated data w/ full trial(s) simulation
* Multiple levels (trials + meta analysis) for all
* Pre-written methods and results section
* Full reproducible data -> manuscript text and fig pipeline
* Pre-populated data w/ full trial(s) simulation
* Multiple levels (trials + meta analysis) for all
March 14, 2025 at 6:25 PM
Reposted by Noah Haber
We are experiencing an assault on science unparalleled by anything I’ve seen in my life. It’s not one issue or another anymore, the entire institution is under attack by the most powerful individuals in the country.
This Friday, where will you be?
standupforscience2025.org
This Friday, where will you be?
standupforscience2025.org
March 2, 2025 at 4:27 PM
We are experiencing an assault on science unparalleled by anything I’ve seen in my life. It’s not one issue or another anymore, the entire institution is under attack by the most powerful individuals in the country.
This Friday, where will you be?
standupforscience2025.org
This Friday, where will you be?
standupforscience2025.org
These two takes are VERY different.
Both identify the severity of the threat we face.
But when it comes to what we should to do about it, one is a call to fight alongside the students and postdocs; the other takes the opportunity to tell those students to stop being critical of university admins.
Both identify the severity of the threat we face.
But when it comes to what we should to do about it, one is a call to fight alongside the students and postdocs; the other takes the opportunity to tell those students to stop being critical of university admins.
Great minds… within 24 hrs both us at @nature.com and @science.org make similar appeals to the research community. And @holdenthorp.bsky.social and I didn’t even exchange notes about it ;-)
🧪
www.nature.com/articles/d41...
www.science.org/doi/10.1126/...
🧪
www.nature.com/articles/d41...
www.science.org/doi/10.1126/...
Come together, right now
The chaos, conflicting information, firings, and hurtful rhetoric of the Trump administration’s approach to science over the past month are causing anxiety, grief, and concern for the scientific commu...
www.science.org
February 27, 2025 at 7:29 PM
These two takes are VERY different.
Both identify the severity of the threat we face.
But when it comes to what we should to do about it, one is a call to fight alongside the students and postdocs; the other takes the opportunity to tell those students to stop being critical of university admins.
Both identify the severity of the threat we face.
But when it comes to what we should to do about it, one is a call to fight alongside the students and postdocs; the other takes the opportunity to tell those students to stop being critical of university admins.
Good news: Added a new use on a slide for how preregistration might enhance research, yay!
Bad news: It's a more robust public record of your research if, say, some capricious and vindictive government decides to pull your funding, force modifications, censors, or cancels it altogether.
Bad news: It's a more robust public record of your research if, say, some capricious and vindictive government decides to pull your funding, force modifications, censors, or cancels it altogether.
February 26, 2025 at 4:40 PM
Good news: Added a new use on a slide for how preregistration might enhance research, yay!
Bad news: It's a more robust public record of your research if, say, some capricious and vindictive government decides to pull your funding, force modifications, censors, or cancels it altogether.
Bad news: It's a more robust public record of your research if, say, some capricious and vindictive government decides to pull your funding, force modifications, censors, or cancels it altogether.
The practice of science is rejecting and challenging authority, not being obedient to it.
Science is punk.
Science is punk.
February 25, 2025 at 9:31 PM
The practice of science is rejecting and challenging authority, not being obedient to it.
Science is punk.
Science is punk.
Reposted by Noah Haber
The problem is that some data always lie while some always tell the truth
February 25, 2025 at 2:43 AM
The problem is that some data always lie while some always tell the truth
Absolutely not. This time calls for an array of approaches, righteous (and just plain right) fury included.
This take is somehow simultaneously entirely predictable and surprising. It's profoundly demoralizing and ineffective, maybe harmful.
www.science.org/doi/10.1126/...
This take is somehow simultaneously entirely predictable and surprising. It's profoundly demoralizing and ineffective, maybe harmful.
www.science.org/doi/10.1126/...
February 25, 2025 at 3:27 PM
Absolutely not. This time calls for an array of approaches, righteous (and just plain right) fury included.
This take is somehow simultaneously entirely predictable and surprising. It's profoundly demoralizing and ineffective, maybe harmful.
www.science.org/doi/10.1126/...
This take is somehow simultaneously entirely predictable and surprising. It's profoundly demoralizing and ineffective, maybe harmful.
www.science.org/doi/10.1126/...
Reposted by Noah Haber
This is fair, but remember if this is something new to you, or just becoming apparent, there's a wide body of research, activism, resistance, stories and archives to draw on to make your case when challenging weak institutions and encouraging colleagues and friends to join you.
If we have any hope of preventing the collapse of US scientific and research environment, we also have to rip off our blinders on the ways our institutions are weak and how they have failed so many.
Those weaknesses are points of leverage for people trying to tear the whole thing down.
Those weaknesses are points of leverage for people trying to tear the whole thing down.
February 16, 2025 at 6:00 PM
This is fair, but remember if this is something new to you, or just becoming apparent, there's a wide body of research, activism, resistance, stories and archives to draw on to make your case when challenging weak institutions and encouraging colleagues and friends to join you.
If we have any hope of preventing the collapse of US scientific and research environment, we also have to rip off our blinders on the ways our institutions are weak and how they have failed so many.
Those weaknesses are points of leverage for people trying to tear the whole thing down.
Those weaknesses are points of leverage for people trying to tear the whole thing down.
February 16, 2025 at 5:38 PM
If we have any hope of preventing the collapse of US scientific and research environment, we also have to rip off our blinders on the ways our institutions are weak and how they have failed so many.
Those weaknesses are points of leverage for people trying to tear the whole thing down.
Those weaknesses are points of leverage for people trying to tear the whole thing down.
When you are imagining a plucky smart investigator epidemiologist going to who knows where to investigate an outbreak like in the movies, you are thinking about the Epidemic Intelligence Service (EIS).
I would strongly recommend avoiding the potato salad for a long while.
I would strongly recommend avoiding the potato salad for a long while.
BREAKING: About 5,200 staff are being laid off across health agencies.
That includes 1,250 people at CDC, including all first-year EIS officers — “disease detectives.”
Employees are bracing for more layoffs in coming days.
That includes 1,250 people at CDC, including all first-year EIS officers — “disease detectives.”
Employees are bracing for more layoffs in coming days.
February 14, 2025 at 7:25 PM
When you are imagining a plucky smart investigator epidemiologist going to who knows where to investigate an outbreak like in the movies, you are thinking about the Epidemic Intelligence Service (EIS).
I would strongly recommend avoiding the potato salad for a long while.
I would strongly recommend avoiding the potato salad for a long while.