My front yard :)

Definitely something worth digging into. I’ll give it some thought

here was our call for methodological standards in metaresearch four years ago. instead of getting fixated on a particular inference we'd like to make, we need to maintain scientific standards, do the hard work, respect the evidence. we can't keep jumping at self-serving solutions without question.

I would add that while pcurve is comprised of tests, that indeed correspond to error rates, the actual hypotheses being tested have little to do with “evidential value”

Test statistics being used are just simple sums, no 3rd moment information enters the test.

Would this be grounds for dismissing the remaining 54 studies as lacking value? It makes no sense. Part of the problem is that the original p curve papers aren’t clear on what exactly is being tested. The authors claim they are tests of skew, but this is incorrect as the

Happy to clarify. Pcurve is used to test whether a set of studies have (or lack) “evidential value” (which is not really defined). But the actual hypotheses being tested by pcurve don’t permit this, as Richard and I show. Suppose one study WAS underpowered in a set of 55 studies -

All pcurve tests are just a simple sum of transformed pvalues. There is a fundamental disconnect between the null hypotheses being tested by p-curve and the claims being made.

What this means is that a significant result for either test only allows one to claim that “at least one” study (out of the set) doesn’t have the property being considered. Why does this happen? Because pcurve completely ignores the configuration of the pvalues being considered.

The test for evidential value simply examines whether the effect size is zero for all studies. The test for lack of evidential value tests whether all studies are “underpowered”, I.e., have small non-centrality parameters.

The developers of p-curve claim that p-curve can be used to make claims about the evidential value (or lack thereof) of whole sets of studies. We show that the actual hypotheses being tested do not allow for such strong conclusions.

The basic idea of p-curve rests on the idea that the skew of a set of p-values is informative about whether QRPs are occurring. As we show, the p-curve tests have nothing to do with skew. It is trivial to create left skewed pvalues that p-curve would confidently label as right skewed.

I'm so sorry this happened to you. There is no excuse for such bs.

Great pic