Ben Recht
@beenwrekt.bsky.social
That 2nd one is not pre-registered RCT z-scores. It's z-scores scraped from the Cochrane Library database. Moreover, why is that one good and the other one bad?
November 16, 2025 at 12:18 AM
That 2nd one is not pre-registered RCT z-scores. It's z-scores scraped from the Cochrane Library database. Moreover, why is that one good and the other one bad?
From a recent talk of mine.
November 15, 2025 at 9:43 PM
From a recent talk of mine.
I’ll take a look, but what makes medicine different from the rest of the human facing sciences is *there are* magic bullets.
November 15, 2025 at 8:24 PM
I’ll take a look, but what makes medicine different from the rest of the human facing sciences is *there are* magic bullets.
Could you explain a bit more what you mean by the 'motivating effect'? Why is it problematic in your mind?
November 15, 2025 at 6:42 PM
Could you explain a bit more what you mean by the 'motivating effect'? Why is it problematic in your mind?
Yes, I agree it's an artifact that a threshold for publication is often "some p-value < 0.05." But in that case, I'd argue the distribution is exactly what I'd expect.
I wrote a bit about it here: www.argmin.net/p/milton-fri...
I wrote a bit about it here: www.argmin.net/p/milton-fri...
Milton Friedman's p-values
Remind me what happens when a measure becomes a target.
www.argmin.net
November 15, 2025 at 6:40 PM
Yes, I agree it's an artifact that a threshold for publication is often "some p-value < 0.05." But in that case, I'd argue the distribution is exactly what I'd expect.
I wrote a bit about it here: www.argmin.net/p/milton-fri...
I wrote a bit about it here: www.argmin.net/p/milton-fri...
Yes. We can definitely say something non-gaussian is going on. But what is it? And how can we make normative claims that it's bad?
Most of the time, people point to this histogram as evidence of bad or questionable research practices. I want to know what backs such reactions.
Most of the time, people point to this histogram as evidence of bad or questionable research practices. I want to know what backs such reactions.
November 15, 2025 at 5:51 PM
Yes. We can definitely say something non-gaussian is going on. But what is it? And how can we make normative claims that it's bad?
Most of the time, people point to this histogram as evidence of bad or questionable research practices. I want to know what backs such reactions.
Most of the time, people point to this histogram as evidence of bad or questionable research practices. I want to know what backs such reactions.
Thanks for sharing.
November 15, 2025 at 5:28 PM
Thanks for sharing.
I think the hard part there is pinning down how the interventions and outcomes are selected. I don't even think you can come up with a well-posed probability distribution over "interventions." What's the sigma-algebra?
November 15, 2025 at 5:26 PM
I think the hard part there is pinning down how the interventions and outcomes are selected. I don't even think you can come up with a well-posed probability distribution over "interventions." What's the sigma-algebra?
And I appreciate you, Berna!
November 15, 2025 at 5:25 PM
And I appreciate you, Berna!
Yes, I've read van Zwet's papers before. He wasn't happy with my reading.
If you will, "fitting a mixture model" and explaining it with a just-so story is a questionable research practice.
If you will, "fitting a mixture model" and explaining it with a just-so story is a questionable research practice.
November 15, 2025 at 5:17 PM
Yes, I've read van Zwet's papers before. He wasn't happy with my reading.
If you will, "fitting a mixture model" and explaining it with a just-so story is a questionable research practice.
If you will, "fitting a mixture model" and explaining it with a just-so story is a questionable research practice.
I don't think this is a good model of science.
November 15, 2025 at 5:15 PM
I don't think this is a good model of science.
This "reasonable" assumption is stating that people are doing random experiments with controls. It's a very comfortable model for statisticians, but it's not only untestable, it's condescending and dismissive.
November 15, 2025 at 5:14 PM
This "reasonable" assumption is stating that people are doing random experiments with controls. It's a very comfortable model for statisticians, but it's not only untestable, it's condescending and dismissive.
Let us now design a meta-scientific null hypothesis significance test to distinguish between Neyman's null and Fisher's null.
November 15, 2025 at 4:11 PM
Let us now design a meta-scientific null hypothesis significance test to distinguish between Neyman's null and Fisher's null.
I mean, a world where papers are only accepted if p<0.05. I don't see this as mysterious or even particularly interesting.
www.argmin.net/p/milton-fri...
www.argmin.net/p/milton-fri...
Milton Friedman's p-values
Remind me what happens when a measure becomes a target.
www.argmin.net
November 15, 2025 at 4:09 PM
I mean, a world where papers are only accepted if p<0.05. I don't see this as mysterious or even particularly interesting.
www.argmin.net/p/milton-fri...
www.argmin.net/p/milton-fri...
That's right. But if I had a generative model for z-scores of medical interventions with mean 0 and variance 1, then all approvals would be false positives.
November 15, 2025 at 3:59 PM
That's right. But if I had a generative model for z-scores of medical interventions with mean 0 and variance 1, then all approvals would be false positives.
That clinical trial plot suggests to me that, on average, nothing in medicine works. Which *coughs* confirms my priors.
The amazing thing about medicine is that almost nothing works, but we find 5-sigma interventions with striking frequency.
The amazing thing about medicine is that almost nothing works, but we find 5-sigma interventions with striking frequency.
November 15, 2025 at 3:52 PM
That clinical trial plot suggests to me that, on average, nothing in medicine works. Which *coughs* confirms my priors.
The amazing thing about medicine is that almost nothing works, but we find 5-sigma interventions with striking frequency.
The amazing thing about medicine is that almost nothing works, but we find 5-sigma interventions with striking frequency.
If the z-scores of randomized experiments were normally distributed, then null-hypothesis-significance-testing-driven science would be an exercise in regression to the mean.
November 15, 2025 at 3:50 PM
If the z-scores of randomized experiments were normally distributed, then null-hypothesis-significance-testing-driven science would be an exercise in regression to the mean.
No, I'm genuinely confused about why z-scores should follow a normal distribution. It seems to me that if that's true, then all statistically significant results should regress to the mean post-experiment.
November 15, 2025 at 3:41 PM
No, I'm genuinely confused about why z-scores should follow a normal distribution. It seems to me that if that's true, then all statistically significant results should regress to the mean post-experiment.
What is publication bias?
November 15, 2025 at 3:34 PM
What is publication bias?
Link to the post if you want to read it. statmodeling.stat.columbia.edu/2025/11/14/t...
The fifth anniversary of a viral histogram | Statistical Modeling, Causal Inference, and Social Science
statmodeling.stat.columbia.edu
November 15, 2025 at 3:31 PM
Link to the post if you want to read it. statmodeling.stat.columbia.edu/2025/11/14/t...
I have attained inner calm, so it's all good.
November 14, 2025 at 8:13 PM
I have attained inner calm, so it's all good.
Thank you... (I think?)
November 14, 2025 at 7:46 PM
Thank you... (I think?)
Link please? I have so much FOMO right now...
November 14, 2025 at 7:33 PM
Link please? I have so much FOMO right now...