We’re all testing for replicability
So far it appears to be a success
So far it appears to be a success
November 7, 2025 at 7:14 PM
We’re all testing for replicability
So far it appears to be a success
So far it appears to be a success
I'd like to re-up this paper we published last year bec I believe it makes a fundamental contribution to theoretical metascience but it is woefully underappreciated. We address a key challenge in estimating the reproducibility of a result: The distance of a replication study from the original. 1/n
November 5, 2025 at 7:07 PM
I'd like to re-up this paper we published last year bec I believe it makes a fundamental contribution to theoretical metascience but it is woefully underappreciated. We address a key challenge in estimating the reproducibility of a result: The distance of a replication study from the original. 1/n
I think this is pretty much what Erkan and I have been working toward for years. Several of our papers establish that as a theoretical. It's a function of three things: the features of the original experiment, the distance of the replication from the original, and the measure of replicability.
November 5, 2025 at 3:12 AM
I think this is pretty much what Erkan and I have been working toward for years. Several of our papers establish that as a theoretical. It's a function of three things: the features of the original experiment, the distance of the replication from the original, and the measure of replicability.
The replicability crisis is getting out of control
October 30, 2025 at 12:34 PM
The replicability crisis is getting out of control
NOW OUT ON FV!!
Enhancing #Transparency & #Replicability in #Data #Collection: Lessons from the Construction of 3 #Education #Datasets
By @adridelrio.bsky.social, Woseeok Kim, @chknutsen.bsky.social, @anjaneundorf.bsky.social, Agustina S. Paglayan & @eugenian.bsky.social
doi.org/10.1017/S153...
Enhancing #Transparency & #Replicability in #Data #Collection: Lessons from the Construction of 3 #Education #Datasets
By @adridelrio.bsky.social, Woseeok Kim, @chknutsen.bsky.social, @anjaneundorf.bsky.social, Agustina S. Paglayan & @eugenian.bsky.social
doi.org/10.1017/S153...
October 30, 2025 at 10:49 AM
NOW OUT ON FV!!
Enhancing #Transparency & #Replicability in #Data #Collection: Lessons from the Construction of 3 #Education #Datasets
By @adridelrio.bsky.social, Woseeok Kim, @chknutsen.bsky.social, @anjaneundorf.bsky.social, Agustina S. Paglayan & @eugenian.bsky.social
doi.org/10.1017/S153...
Enhancing #Transparency & #Replicability in #Data #Collection: Lessons from the Construction of 3 #Education #Datasets
By @adridelrio.bsky.social, Woseeok Kim, @chknutsen.bsky.social, @anjaneundorf.bsky.social, Agustina S. Paglayan & @eugenian.bsky.social
doi.org/10.1017/S153...
Reminded today of this opinion piece by Meredith Ringel Morris, which I believe introduced the term "prompt hacking" (in the sense of p-hacking rather than jailbreaking) and the central concerns about replicability echoed in later papers.
"Prompting Considered Harmful"
dl.acm.org/doi/full/10....
"Prompting Considered Harmful"
dl.acm.org/doi/full/10....
October 29, 2025 at 7:00 PM
Reminded today of this opinion piece by Meredith Ringel Morris, which I believe introduced the term "prompt hacking" (in the sense of p-hacking rather than jailbreaking) and the central concerns about replicability echoed in later papers.
"Prompting Considered Harmful"
dl.acm.org/doi/full/10....
"Prompting Considered Harmful"
dl.acm.org/doi/full/10....
I think I posted about it before but never with a thread. We recently put a new preprint on arxiv.
📖 Replicable Reinforcement Learning with Linear Function Approximation
🔗 arxiv.org/abs/2509.08660
In this paper, we study formal replicability in RL with linear function approximation. The... (1/6)
📖 Replicable Reinforcement Learning with Linear Function Approximation
🔗 arxiv.org/abs/2509.08660
In this paper, we study formal replicability in RL with linear function approximation. The... (1/6)
Replicable Reinforcement Learning with Linear Function Approximation
Replication of experimental results has been a challenge faced by many scientific disciplines, including the field of machine learning. Recent work on the theory of machine learning has formalized rep...
arxiv.org
October 26, 2025 at 2:16 PM
I think I posted about it before but never with a thread. We recently put a new preprint on arxiv.
📖 Replicable Reinforcement Learning with Linear Function Approximation
🔗 arxiv.org/abs/2509.08660
In this paper, we study formal replicability in RL with linear function approximation. The... (1/6)
📖 Replicable Reinforcement Learning with Linear Function Approximation
🔗 arxiv.org/abs/2509.08660
In this paper, we study formal replicability in RL with linear function approximation. The... (1/6)
A similar argument is made that the discussion about replicability crisis should not be done in public because it undermines trustworthiness. Personally I can't see this as anything other than setting up an external agenda that materially undermines the trustworthiness of science.
October 21, 2025 at 4:15 PM
A similar argument is made that the discussion about replicability crisis should not be done in public because it undermines trustworthiness. Personally I can't see this as anything other than setting up an external agenda that materially undermines the trustworthiness of science.
On the upside INSANE amount of replicability so even for 80 AUD price tag it seems worth it due to amount of monsters. I've played 5 of the monsters a few times each and already felt it was worth the money. If you like BULLET, this takes the same philosophy IMO.
October 19, 2025 at 7:19 AM
On the upside INSANE amount of replicability so even for 80 AUD price tag it seems worth it due to amount of monsters. I've played 5 of the monsters a few times each and already felt it was worth the money. If you like BULLET, this takes the same philosophy IMO.
Carl Sagan distinguished science from faith, despite the lack of certainty inherent in both, by its replicability.
And it applies perfectly to the debate between classical history and materialist analyses of resistance: It will always develop against oppressive forces. Like a law of science.
And it applies perfectly to the debate between classical history and materialist analyses of resistance: It will always develop against oppressive forces. Like a law of science.
“This fact cannot be eliminated from the mind of the popular masses... Sooner or later they will hear our voice… But even if ... the enemies of the people were to conquer us and we were to perish... our memory would still pass from generation to generation and awaken posterity to a new struggle.“
October 17, 2025 at 4:14 PM
Carl Sagan distinguished science from faith, despite the lack of certainty inherent in both, by its replicability.
And it applies perfectly to the debate between classical history and materialist analyses of resistance: It will always develop against oppressive forces. Like a law of science.
And it applies perfectly to the debate between classical history and materialist analyses of resistance: It will always develop against oppressive forces. Like a law of science.
ED: Some studies have been done: Twin studies, candidate gene studies looking at hormone receptors (limited replicability), whole exome sequencing studies (limited by small sample sizes)
Some trans individuals see potential benefits of this research, but other worry about discrimination #ASHG25
Some trans individuals see potential benefits of this research, but other worry about discrimination #ASHG25
October 15, 2025 at 12:40 PM
ED: Some studies have been done: Twin studies, candidate gene studies looking at hormone receptors (limited replicability), whole exome sequencing studies (limited by small sample sizes)
Some trans individuals see potential benefits of this research, but other worry about discrimination #ASHG25
Some trans individuals see potential benefits of this research, but other worry about discrimination #ASHG25
10/11/25 Here’s Your Daily Disinformation Detox Reminder: Today, learn the importance of replicability in ensuring the validity of scientific research. Beware of studies that can’t be replicated or that have failed to replicate. Remember, replicability is a cornerstone of the scientific method.
October 11, 2025 at 11:05 AM
10/11/25 Here’s Your Daily Disinformation Detox Reminder: Today, learn the importance of replicability in ensuring the validity of scientific research. Beware of studies that can’t be replicated or that have failed to replicate. Remember, replicability is a cornerstone of the scientific method.
Yep I agree. The speed and replicability is a differentiator. (As they say, teach a computer to be phished…)
October 9, 2025 at 1:26 PM
Yep I agree. The speed and replicability is a differentiator. (As they say, teach a computer to be phished…)
📢 IMC 2026 CFP is open! Submit measurement research by Nov. 20, 2025, or Apr. 29, 2026.
Two cycles, one-shot revisions, replicability track. Conference in Karlsruhe, Germany 🇩🇪 Nov 2026. Details: conferences.sigcomm.org/imc/2026/cfp/ #IMC2026 #NetworkMeasurement
Two cycles, one-shot revisions, replicability track. Conference in Karlsruhe, Germany 🇩🇪 Nov 2026. Details: conferences.sigcomm.org/imc/2026/cfp/ #IMC2026 #NetworkMeasurement
ACM IMC 2026 - Call for Papers
conferences.sigcomm.org
October 7, 2025 at 10:43 AM
📢 IMC 2026 CFP is open! Submit measurement research by Nov. 20, 2025, or Apr. 29, 2026.
Two cycles, one-shot revisions, replicability track. Conference in Karlsruhe, Germany 🇩🇪 Nov 2026. Details: conferences.sigcomm.org/imc/2026/cfp/ #IMC2026 #NetworkMeasurement
Two cycles, one-shot revisions, replicability track. Conference in Karlsruhe, Germany 🇩🇪 Nov 2026. Details: conferences.sigcomm.org/imc/2026/cfp/ #IMC2026 #NetworkMeasurement
How causality ate the world. I don’t see how causality is the key issue in psychology research, most of which is based on RCTs. Replicability crisis, yes a big issue. Generalizability out of the lab, yes. But causality? That hardly seems to be the crux.
Happy to announce that I'll give a talk on how we can make rigorous causal inference more mainstream 📈
You can sign up for the Zoom link here: tinyurl.com/CIIG-JuliaRo...
You can sign up for the Zoom link here: tinyurl.com/CIIG-JuliaRo...
October 6, 2025 at 2:06 PM
How causality ate the world. I don’t see how causality is the key issue in psychology research, most of which is based on RCTs. Replicability crisis, yes a big issue. Generalizability out of the lab, yes. But causality? That hardly seems to be the crux.
#ESHE2025
Manuel Will ‘Is there a reproducibility crisis in lithic analysis?’
Not enough studies on replicability!
Project on MSA – 38 vars inter-obs replication on exp flks set: pretty good replicability in most, but not all ☹ 0.2 score for scar directionality!!!
Choose traits, train 😍!!
Manuel Will ‘Is there a reproducibility crisis in lithic analysis?’
Not enough studies on replicability!
Project on MSA – 38 vars inter-obs replication on exp flks set: pretty good replicability in most, but not all ☹ 0.2 score for scar directionality!!!
Choose traits, train 😍!!
September 27, 2025 at 9:12 AM
#ESHE2025
Manuel Will ‘Is there a reproducibility crisis in lithic analysis?’
Not enough studies on replicability!
Project on MSA – 38 vars inter-obs replication on exp flks set: pretty good replicability in most, but not all ☹ 0.2 score for scar directionality!!!
Choose traits, train 😍!!
Manuel Will ‘Is there a reproducibility crisis in lithic analysis?’
Not enough studies on replicability!
Project on MSA – 38 vars inter-obs replication on exp flks set: pretty good replicability in most, but not all ☹ 0.2 score for scar directionality!!!
Choose traits, train 😍!!
The replication crisis just isn’t a science wide phenomenon and rates of replicability are utterly meaningless even with psychology. There’s precisely zero empirical evidence that open science can fix the purported problem.
There’s certainly nothing to justify any of this science wide.
There’s certainly nothing to justify any of this science wide.
September 26, 2025 at 9:40 PM
The replication crisis just isn’t a science wide phenomenon and rates of replicability are utterly meaningless even with psychology. There’s precisely zero empirical evidence that open science can fix the purported problem.
There’s certainly nothing to justify any of this science wide.
There’s certainly nothing to justify any of this science wide.
Replicability and transparency in physical therapy research: Time to wake up! doi.org/10.1101/2025...
September 25, 2025 at 8:37 PM
Replicability and transparency in physical therapy research: Time to wake up! doi.org/10.1101/2025...
Folks *still* quote estimates of replicability when sample size was conditioned on the original effect size as though it’s a meaningful number. They still p-curve and claim it is a test that unlocks the file drawer or adjusts for p-hacking. These aren’t valid claims for critique they’re just wrong.
September 25, 2025 at 11:25 AM
Folks *still* quote estimates of replicability when sample size was conditioned on the original effect size as though it’s a meaningful number. They still p-curve and claim it is a test that unlocks the file drawer or adjusts for p-hacking. These aren’t valid claims for critique they’re just wrong.
And not publishing in a peer reviewed journal so the claim can be tested for replicability.
September 22, 2025 at 6:48 PM
And not publishing in a peer reviewed journal so the claim can be tested for replicability.
High time for more articles on the Replicability Crisis in Confident Reckons On What Will Pacify Authoritarian Voters.
September 22, 2025 at 9:41 AM
High time for more articles on the Replicability Crisis in Confident Reckons On What Will Pacify Authoritarian Voters.
If we crack intelligence (huge if) then the thought speed and replicability alone will likely surpass human thought. Once that happens it can invent better AI. I think it’s binary, either we crack it and we’re all out of a job or we don’t.
September 20, 2025 at 9:06 PM
If we crack intelligence (huge if) then the thought speed and replicability alone will likely surpass human thought. Once that happens it can invent better AI. I think it’s binary, either we crack it and we’re all out of a job or we don’t.
Call for Papers: Advances in Methods and Practices in Psychological Science is calling for proposals for manuscripts that discuss, demonstrate, and work to enhance replicability and reproducibility of methods research in psychology.
Proposals are due by September 15!
Proposals are due by September 15!
<em>AMPPS</em> Call for Papers on Replicability and Reproducibility in Methodological Research
Methodological reform has shaped the last decade of psychological research. Researchers have undertaken replication studies, journals and funders have emphasized registration and data sharing, and aut...
www.psychologicalscience.org
September 12, 2025 at 6:59 PM
Call for Papers: Advances in Methods and Practices in Psychological Science is calling for proposals for manuscripts that discuss, demonstrate, and work to enhance replicability and reproducibility of methods research in psychology.
Proposals are due by September 15!
Proposals are due by September 15!
🧬 Learn about the results and implications of the Reproducibility Project: Cancer Biology!
Attend the “Assessing research replicability" lecture by Dr. Tim Errington on Thurs 18 Sept, 15:30-16:15.
👉 Register here: www.pretix.osc.lmu.de/lmu-osc/OSSS...
Attend the “Assessing research replicability" lecture by Dr. Tim Errington on Thurs 18 Sept, 15:30-16:15.
👉 Register here: www.pretix.osc.lmu.de/lmu-osc/OSSS...
September 11, 2025 at 5:44 PM
🧬 Learn about the results and implications of the Reproducibility Project: Cancer Biology!
Attend the “Assessing research replicability" lecture by Dr. Tim Errington on Thurs 18 Sept, 15:30-16:15.
👉 Register here: www.pretix.osc.lmu.de/lmu-osc/OSSS...
Attend the “Assessing research replicability" lecture by Dr. Tim Errington on Thurs 18 Sept, 15:30-16:15.
👉 Register here: www.pretix.osc.lmu.de/lmu-osc/OSSS...
The pretty draft is now online.
Link to paper (free): www.journals.uchicago.edu/doi/epdf/10....
Our replication package starts from the raw data and we put real work into making it readable & setting it up so people could poke at it, so please do explore it: dataverse.harvard.edu/dataset.xhtm...
Link to paper (free): www.journals.uchicago.edu/doi/epdf/10....
Our replication package starts from the raw data and we put real work into making it readable & setting it up so people could poke at it, so please do explore it: dataverse.harvard.edu/dataset.xhtm...
September 10, 2025 at 5:25 PM
The pretty draft is now online.
Link to paper (free): www.journals.uchicago.edu/doi/epdf/10....
Our replication package starts from the raw data and we put real work into making it readable & setting it up so people could poke at it, so please do explore it: dataverse.harvard.edu/dataset.xhtm...
Link to paper (free): www.journals.uchicago.edu/doi/epdf/10....
Our replication package starts from the raw data and we put real work into making it readable & setting it up so people could poke at it, so please do explore it: dataverse.harvard.edu/dataset.xhtm...