Pamela Reinagel
@neurotheory.bsky.social
Theoretical and Computational Neuroscientist at UCSD www.ratrix.org. Neural coding, natural scene statistics, visual behavior, value-based decision-making, statistical & non-statistical inference, history & philosophy of science, research rigor, pedagogy.
#VisionScience community - please repost and share with trainees who might be interested in the job!
November 17, 2023 at 12:22 AM
#VisionScience community - please repost and share with trainees who might be interested in the job!
Sure. Understanding the effects of each quantitatively and contextually aids interpretation of metascience data, e.g., the likely impact on past literature where a practice has been used; and helps focus attention and education on those practices that are actually responsible for the most harm.
November 2, 2023 at 3:16 PM
Sure. Understanding the effects of each quantitatively and contextually aids interpretation of metascience data, e.g., the likely impact on past literature where a practice has been used; and helps focus attention and education on those practices that are actually responsible for the most harm.
I do mention sequential analysis, and cite your fine paper and others, for those who want to learn more. If my simulation reflects their existing practice, prespecifying what they already do may be their preferred choice, and is valid. Bonferroni kills power so it would be a particularly bad choice.
November 2, 2023 at 3:11 PM
I do mention sequential analysis, and cite your fine paper and others, for those who want to learn more. If my simulation reflects their existing practice, prespecifying what they already do may be their preferred choice, and is valid. Bonferroni kills power so it would be a particularly bad choice.
Others may think "I had no idea reporting p values constrained me in such a way! I shouldn't be reporting p values." A difference in our perspective is, I work in fields where most work is not even intended to be confirmatory, people just give p values because they're told to, and don't know better.
November 2, 2023 at 3:07 PM
Others may think "I had no idea reporting p values constrained me in such a way! I shouldn't be reporting p values." A difference in our perspective is, I work in fields where most work is not even intended to be confirmatory, people just give p values because they're told to, and don't know better.
Actually I think I have a higher opinion of my readers than you; I don't think they'll glibly walk away with a superficial take. Some will come away with: "I really care about controlling false positives, but cool, I could get more flexibility and statistical power with sequential analysis."
November 2, 2023 at 3:04 PM
Actually I think I have a higher opinion of my readers than you; I don't think they'll glibly walk away with a superficial take. Some will come away with: "I really care about controlling false positives, but cool, I could get more flexibility and statistical power with sequential analysis."
Important difference - this would be w=19 in my simulations, (my Fig1). I claim FP<α(1+w/2) or <0.525 which accords. They got much lower than this bc they limited the maximum sample size. I focused on effect of limiting w (only adding data if p is near alpha) which I think reflects common practice.
November 2, 2023 at 2:58 PM
Important difference - this would be w=19 in my simulations, (my Fig1). I claim FP<α(1+w/2) or <0.525 which accords. They got much lower than this bc they limited the maximum sample size. I focused on effect of limiting w (only adding data if p is near alpha) which I think reflects common practice.
Bottom line: collecting more data to shore up a finding isn’t bad science, it’s just bad for p-values. My purpose is not to justify or encourage p-hacking, but rather to bring to light some poorly-appreciated facts, and enable more informed and transparent choices. Plus, it was a fun puzzle. [3/3]
November 1, 2023 at 11:45 PM
Bottom line: collecting more data to shore up a finding isn’t bad science, it’s just bad for p-values. My purpose is not to justify or encourage p-hacking, but rather to bring to light some poorly-appreciated facts, and enable more informed and transparent choices. Plus, it was a fun puzzle. [3/3]
With the goal of teaching the perils of N-hacking, I simulated a realistic lab situation. To my surprise, the increase in false positives was slight. Moreover, N-hacking increased the chance a result would be replicable (the PPV). This paper shows why and when this is the case. [2/3]
November 1, 2023 at 11:42 PM
With the goal of teaching the perils of N-hacking, I simulated a realistic lab situation. To my surprise, the increase in false positives was slight. Moreover, N-hacking increased the chance a result would be replicable (the PPV). This paper shows why and when this is the case. [2/3]
If a study finds an effect that is not significant, it is considered a "questionable research practice" to collect more data to reach significance. This kind of p-hacking is often cited as a cause of unreproducible results. This is troubling, as the practice is common in biology. [1/3]
November 1, 2023 at 11:41 PM
If a study finds an effect that is not significant, it is considered a "questionable research practice" to collect more data to reach significance. This kind of p-hacking is often cited as a cause of unreproducible results. This is troubling, as the practice is common in biology. [1/3]
I think we have run up against the limitations of this medium. I actually have no idea what you meant by that post. But what I take away is we are interested in a related question, we appear to have different views and referents, and it would be worth digging into in some less staccato format.
October 23, 2023 at 4:31 AM
I think we have run up against the limitations of this medium. I actually have no idea what you meant by that post. But what I take away is we are interested in a related question, we appear to have different views and referents, and it would be worth digging into in some less staccato format.
Using methods of measurement whose mechanisms are well understood, thus whose assumptions and limitations are front-of-mind; also frequent use of triangulation: testing a casual model or theory by many distinct techniques that have different assumptions… I can give examples but not briefly. (2/2)
October 22, 2023 at 9:09 PM
Using methods of measurement whose mechanisms are well understood, thus whose assumptions and limitations are front-of-mind; also frequent use of triangulation: testing a casual model or theory by many distinct techniques that have different assumptions… I can give examples but not briefly. (2/2)
it’s hard to articulate examples in a few words, as a key characteristic is not thinking “science“ can ever be a short, simple, one-shot thing. It takes a long series of purposeful observations, theory, highly constrained models that make specific predictions, many many controlled experiments, (1/2)
October 22, 2023 at 9:01 PM
it’s hard to articulate examples in a few words, as a key characteristic is not thinking “science“ can ever be a short, simple, one-shot thing. It takes a long series of purposeful observations, theory, highly constrained models that make specific predictions, many many controlled experiments, (1/2)
looks interesting but can’t tell from a quick look if it’s the same methodology I had in mind, or one that includes both sides if the debate we were having, or something else entirely
October 22, 2023 at 8:53 PM
looks interesting but can’t tell from a quick look if it’s the same methodology I had in mind, or one that includes both sides if the debate we were having, or something else entirely
There may be other reasons to study what scientists do in fields with poor track records, or fields with no track record,
But I think Phil of Sci aimed at improving practice can be advanced by studying what *good* or *great* scientists actually do, if you can identify who those are.
But I think Phil of Sci aimed at improving practice can be advanced by studying what *good* or *great* scientists actually do, if you can identify who those are.
October 22, 2023 at 5:22 PM
There may be other reasons to study what scientists do in fields with poor track records, or fields with no track record,
But I think Phil of Sci aimed at improving practice can be advanced by studying what *good* or *great* scientists actually do, if you can identify who those are.
But I think Phil of Sci aimed at improving practice can be advanced by studying what *good* or *great* scientists actually do, if you can identify who those are.
Of course there is a normative judgement there— these scientific fields have been incredibly successful at generating secure knowledge (discoveries and that stood the test of time, replicated endlessly, generalized broadly, parsimoniously explained much, led to successful practical applications…)
October 22, 2023 at 5:19 PM
Of course there is a normative judgement there— these scientific fields have been incredibly successful at generating secure knowledge (discoveries and that stood the test of time, replicated endlessly, generalized broadly, parsimoniously explained much, led to successful practical applications…)
Opposite perspective. As one trained in classical biochemistry, genetics and (early/foundational) molecular biology, I think scientists in these fields have/had some brilliant, innovative, rigorous and productive methods that are not yet well codified, nor yet understood in philosophy or statistics
October 22, 2023 at 5:14 PM
Opposite perspective. As one trained in classical biochemistry, genetics and (early/foundational) molecular biology, I think scientists in these fields have/had some brilliant, innovative, rigorous and productive methods that are not yet well codified, nor yet understood in philosophy or statistics
P.S. examples: some biochemistry, molecular biology, basic research on experimental model organisms. E.g., almost no p values in the landmark papers of molecular biology (Meselson & Stahl, Hershey & Chase, Jacob & Monod, Nirenberg & Matthaei... not even the overtly statistical Luria & Delbruck).
September 25, 2023 at 4:39 PM
P.S. examples: some biochemistry, molecular biology, basic research on experimental model organisms. E.g., almost no p values in the landmark papers of molecular biology (Meselson & Stahl, Hershey & Chase, Jacob & Monod, Nirenberg & Matthaei... not even the overtly statistical Luria & Delbruck).
the rigor and reliability of the research could be very high, because the p values were not the basis for the conclusions. I think eliminating the reporting of performative p values is very important. Otherwise people outside the field are misled about the epistemic basis and status of the claims.
September 25, 2023 at 4:12 PM
the rigor and reliability of the research could be very high, because the p values were not the basis for the conclusions. I think eliminating the reporting of performative p values is very important. Otherwise people outside the field are misled about the epistemic basis and status of the claims.
If you ask them what the p value does tell them, they mostly think it is the PPV. This is what they want to know. So with better education they probably wouldn't use p values, and might use Bayesian statistics. In such subfields most p values in the literature are invalid (post hoc), and yet [5/6]
September 25, 2023 at 4:06 PM
If you ask them what the p value does tell them, they mostly think it is the PPV. This is what they want to know. So with better education they probably wouldn't use p values, and might use Bayesian statistics. In such subfields most p values in the literature are invalid (post hoc), and yet [5/6]
Some scientists compute p values as an afterthought, after drawing conclusions, while writing papers. They do it because they think it is expected or required. But often they have strong justifications for their conclusions, which are given in the paper, and which are what sways their readers. [4/n]
September 25, 2023 at 4:01 PM
Some scientists compute p values as an afterthought, after drawing conclusions, while writing papers. They do it because they think it is expected or required. But often they have strong justifications for their conclusions, which are given in the paper, and which are what sways their readers. [4/n]
I asked colleagues how much they rely on p values to decide if they believe something. The only ones who said p was important were trained in psychology (and they said it was the ONLY reason to believe something). Which is interesting. Psychology's excessive reliance may have caused their woes [3/n]
September 25, 2023 at 3:51 PM
I asked colleagues how much they rely on p values to decide if they believe something. The only ones who said p was important were trained in psychology (and they said it was the ONLY reason to believe something). Which is interesting. Psychology's excessive reliance may have caused their woes [3/n]