Completely agree with both of these sentiments — setting a meaningful SESOI is hard, but it’s the right discussion to have!

bsky.app/profile/anne...

If I understand you correctly, that would imply restricting your sample size/power so that effects smaller than what you consider relevant wouldn’t become significant. That’s not ideal. Equivalence tests allow you to distinguish between significant and relevant.

I personally think that dichotomous claims are very important in science (to the degree that I think binary statistical decision criteria would keep re-emerging naturally even if you’d somehow abolish them now) but of course there are many research questions for which tests would be the wrong tool.

1) IMO the choice between Bayesian vs frequentist methods is orthogonal to the choice between tests and (eg) estimation. 2) If you’re not interested in dichotomous claims, of course you don’t need to test, which I tried to imply here bsky.app/profile/anne...

So (beyond this specific example of anchor-based approaches) I would be happy to see many diverse applied examples on how SESOIs have been reasonably specified in sport and exercise science (if they exist...).

(Or that a test actually isn’t the best tool for answering the research question, for that matter)

Jinx!

These are exactly the right discussions to have IMO, especially when they help us better understand what knowledge we need (and might still be missing) for setting up meaningful, informative tests.

I completely agree! But if you realise that you can’t specify a SESOI for the life of you, it usually means that you can’t specify your research hypothesis enough to make it statistically testable. In that case, the best decision may to do something else: journals.sagepub.com/doi/10.1177/...

Without an explicit SESOI, readers will apply implicit SESOIs (as happened here) and the discussion becomes unnecessarily confused. SESOIs have nothing to do with sample size, but "with large N, people suddenly become aware of the matter" (@dingdingpeng.the100.ci). Thanks for coming to my TED talk.

The solution is to specify and test your alternative hypothesis. You can do this by defining a smallest effect size of interest (SESOI) and performing an equivalence or inferiority test, *in addition to* the null-hypothesis test. Here's a whole tutorial about it: doi.org/10.1177/2515... >

But *only* testing against 0 usually isn't very interesting and has problems. If you get a non-sig. result, you can claim that power was too low and dismiss it. If you get a sig. result but the effect size is very small, @aaronjfisher.bsky.social may come and say that it's actually meaningless. >

Statistical tests (and thus p-values, if you use frequentist methods) are useful if you want to test a hypothesis or make a dichotomous claim (for a nice overview, see doi.org/10.1177/0959... by @uyguntunc.bsky.social et al.), regardless of whether N is small or large. >

2) But the real elephant in the room is *your effect of interest*. @aaronjfisher.bsky.social is right in that statistical significance doesn't tell you if an effect is practically or theoretically meaningful. But that neither means that p-values are useless, nor does it depend on the sample size. >

5% is quite a lot if you think about it. Huge N gives you the luxury to reduce alpha by a lot and still keep very high power. E.g., alpha = 0.5% (0.005) would give you 98.8% power for the same effect (in a t-test). The best balance depends on the cost of each error type, see tinyurl.com/yut35b3u >

1) With the conventional alpha = 5% and a huge sample, you may have extremely high power for your effect of interest – say, 99.9%. That means beta (type-II error rate) = 0.1%. Are you sure that you want your type-I error rate to be 50x the size of your type-II error rate? >