While these insights likely won’t differ drastically across businesses, performing such analysis with your own experiments can help clarify the (implicit) trade-offs you're making.

mihagazvoda.com/files/nhst-r...

6/6

To address these, a useful alternative I see is applying a shrinkage estimator based on past experiments (and experiment-specific details), combined with explicit cost-benefit analysis. As @statmodeling.bsky.social puts it, let’s make decisions in real-world units: dollars, customers.

5/6

* Sacrifices business impact by excessively focusing on minimizing losses at the expense of gains.
* Misleads learnings due to exaggerated significant effect estimates, poor coverage, and low replication rates—though the estimated direction typically remains correct.

4/6

Adopting this approach reveals that, under realistic assumptions, NHST with conventional error rates often:

3/6

I highlight how NHST’s effectiveness is often explained using an unrealistic scenario—pretending effects are strictly either “null” or “true.” Instead, I suggest performing a meta-analysis of past experiments to estimate the distribution of unobserved true effects.

2/6

I like this paper about CI interpretation: pure.rug.nl/ws/portalfil...

Gelman, Hill, and Vehtari in Regression and Other Stories suggest fitting a regression with the same variables as were used for matching/IPTW.