Joris Frese
@fresejoris.bsky.social
PhD candidate in political science at the EUI [On the 25/26 job market].
Currently visiting Harvard's Department of Government.
Interested in: political behavior, quantitative methods, metascience.
https://www.jorisfrese.com/
Currently visiting Harvard's Department of Government.
Interested in: political behavior, quantitative methods, metascience.
https://www.jorisfrese.com/
Delighted to be visiting Harvard's Department of Government as a Fellow. Loving the free food at most seminars here (EUI take note)! I have an office at the CGIS during the Fall and Spring Semester, so if you are around and would like to talk about research or play a game of chess, let me know 🙂
November 7, 2025 at 1:10 PM
Delighted to be visiting Harvard's Department of Government as a Fellow. Loving the free food at most seminars here (EUI take note)! I have an office at the CGIS during the Fall and Spring Semester, so if you are around and would like to talk about research or play a game of chess, let me know 🙂
Check out Miriam's great thread on our new nano-targeting paper below. One of the most innovative projects I ever had the privilege of contributing to!
October 16, 2025 at 8:08 AM
Check out Miriam's great thread on our new nano-targeting paper below. One of the most innovative projects I ever had the privilege of contributing to!
I personally have used z-curves in a meta-analytic paper of mine: journals.sagepub.com/doi/10.1177/.... See the brief excerpt of my appendix where I give some reasons why I prefer them over p-curves. The cited studies by Bartos, Brunner, and Schimmack go into much more detail on this.
August 9, 2025 at 6:59 AM
I personally have used z-curves in a meta-analytic paper of mine: journals.sagepub.com/doi/10.1177/.... See the brief excerpt of my appendix where I give some reasons why I prefer them over p-curves. The cited studies by Bartos, Brunner, and Schimmack go into much more detail on this.
In the brief excerpts of the Table here, you may already identify several metrics that could be more suitable for the stylized examples described earlier in this thread (compared to a simple significance test), such as meta-analytic tests for (i) and tests based on effect sizes for (ii). 12/14
July 21, 2025 at 8:41 AM
In the brief excerpts of the Table here, you may already identify several metrics that could be more suitable for the stylized examples described earlier in this thread (compared to a simple significance test), such as meta-analytic tests for (i) and tests based on effect sizes for (ii). 12/14
There are lots of informative summary statistics on all these parameters in the review, but the heart and the main contribution of our paper is Table 4, where we offer an overview of each of the 50 different reproducibility metrics and their use-cases. 11/14
July 21, 2025 at 8:41 AM
There are lots of informative summary statistics on all these parameters in the review, but the heart and the main contribution of our paper is Table 4, where we offer an overview of each of the 50 different reproducibility metrics and their use-cases. 11/14
Our initial search strings resulted in more than 1300 potentially relevant papers to be screened (and an additional 4000+ from their reference lists). After further screening, our list of the most relevant papers included 49 large-scale replication papers and 95 methods papers. 8/14
July 21, 2025 at 8:41 AM
Our initial search strings resulted in more than 1300 potentially relevant papers to be screened (and an additional 4000+ from their reference lists). After further screening, our list of the most relevant papers included 49 large-scale replication papers and 95 methods papers. 8/14
🇨🇭🇪🇺 Just Published in Royal Society Open Science!
A scoping review on metrics to quantify reproducibility:
royalsocietypublishing.org/doi/10.1098/...
Ever conducted a replication and pondered when/how to conclude if it was (un)successful?
We have just the paper for you (led by Rachel Heyard)! 1/14
A scoping review on metrics to quantify reproducibility:
royalsocietypublishing.org/doi/10.1098/...
Ever conducted a replication and pondered when/how to conclude if it was (un)successful?
We have just the paper for you (led by Rachel Heyard)! 1/14
July 21, 2025 at 8:41 AM
🇨🇭🇪🇺 Just Published in Royal Society Open Science!
A scoping review on metrics to quantify reproducibility:
royalsocietypublishing.org/doi/10.1098/...
Ever conducted a replication and pondered when/how to conclude if it was (un)successful?
We have just the paper for you (led by Rachel Heyard)! 1/14
A scoping review on metrics to quantify reproducibility:
royalsocietypublishing.org/doi/10.1098/...
Ever conducted a replication and pondered when/how to conclude if it was (un)successful?
We have just the paper for you (led by Rachel Heyard)! 1/14
Lastly, we use uesdRobust to replicate the two most highly-cited UESD studies published in the APSR. We find that some of their estimates lost conventional statistical significance once error inflation is adjusted for, while others remain placebo-robust. 9/9
July 1, 2025 at 2:44 PM
Lastly, we use uesdRobust to replicate the two most highly-cited UESD studies published in the APSR. We find that some of their estimates lost conventional statistical significance once error inflation is adjusted for, while others remain placebo-robust. 9/9
To implement this adjustment procedure, we present our novel R package uesdRobust, which serves to estimate conventional and placebo-robust UESDs (OLS & RDD) and produces easily interpretable summary outputs. See some examples of unaltered package outputs attached. 7/9
July 1, 2025 at 2:44 PM
To implement this adjustment procedure, we present our novel R package uesdRobust, which serves to estimate conventional and placebo-robust UESDs (OLS & RDD) and produces easily interpretable summary outputs. See some examples of unaltered package outputs attached. 7/9
By comparing how extreme ones UESD test statistic is compared to the distribution of all placebo test statistics in a given context, one can restore the nominal 5% error rate even in the presence of substantial error inflation across unadjusted estimates. 6/9
July 1, 2025 at 2:44 PM
By comparing how extreme ones UESD test statistic is compared to the distribution of all placebo test statistics in a given context, one can restore the nominal 5% error rate even in the presence of substantial error inflation across unadjusted estimates. 6/9
This troubling finding holds across the whole range of all common UESD specifications and even when using outcome variables that are conceptually impossible to be affected by unexpected events. There is also substantial heterogeneity in error rates across contexts. 4/9
July 1, 2025 at 2:44 PM
This troubling finding holds across the whole range of all common UESD specifications and even when using outcome variables that are conceptually impossible to be affected by unexpected events. There is also substantial heterogeneity in error rates across contexts. 4/9
..we first conduct a large-scale placebo exercise using ~1 million surveys completed over 20 years across 40 countries to determine the false-positive rate in UESD studies. We find that ~10% of all placebo tests produce significant estimates, meaning type I errors are inflated by a factor of 2.
3/9
3/9
July 1, 2025 at 2:44 PM
..we first conduct a large-scale placebo exercise using ~1 million surveys completed over 20 years across 40 countries to determine the false-positive rate in UESD studies. We find that ~10% of all placebo tests produce significant estimates, meaning type I errors are inflated by a factor of 2.
3/9
3/9
The UESD is now a well-established causal inference method and UESD studies regularly get published in all the top political science journals and beyond. But is this literature robust? To assess this… 2/9
July 1, 2025 at 2:44 PM
The UESD is now a well-established causal inference method and UESD studies regularly get published in all the top political science journals and beyond. But is this literature robust? To assess this… 2/9
New WP w/ @riazsascha.bsky.social: via >42 million placebo tests, we show type I error inflation in Unexpected Event during Survey Designs (UESD), propose an adjustment, implement via new R package uesdRobust, and demonstrate utility by replicating two top UESD studies.
osf.io/preprints/os...
1/9
osf.io/preprints/os...
1/9
July 1, 2025 at 2:44 PM
New WP w/ @riazsascha.bsky.social: via >42 million placebo tests, we show type I error inflation in Unexpected Event during Survey Designs (UESD), propose an adjustment, implement via new R package uesdRobust, and demonstrate utility by replicating two top UESD studies.
osf.io/preprints/os...
1/9
osf.io/preprints/os...
1/9
Furthermore, unlike many other Unexpected Event effects, this effect was actually not short-lived: the improvements in immigration attitudes lasted for several months and the post-takeover time trend in attitudes is significantly more positive than the pre-takeover slope. 7/8
May 21, 2025 at 10:03 AM
Furthermore, unlike many other Unexpected Event effects, this effect was actually not short-lived: the improvements in immigration attitudes lasted for several months and the post-takeover time trend in attitudes is significantly more positive than the pre-takeover slope. 7/8
Indeed, this is what I find, using a regression discontinuity design around the Fall of Kabul: in the immediate aftermath of this event, Europeans became significantly and substantially more supportive of immigration. 6/8
May 21, 2025 at 10:03 AM
Indeed, this is what I find, using a regression discontinuity design around the Fall of Kabul: in the immediate aftermath of this event, Europeans became significantly and substantially more supportive of immigration. 6/8
I also show that this increased salience was accompanied by positive changes to the framing of immigration: the sentiment of Tweets and news articles about immigrants became significantly more positive, because refugees fleeing the Taliban were consistently framed as deserving of help. 4/8
May 21, 2025 at 10:03 AM
I also show that this increased salience was accompanied by positive changes to the framing of immigration: the sentiment of Tweets and news articles about immigrants became significantly more positive, because refugees fleeing the Taliban were consistently framed as deserving of help. 4/8
Analyzing social media and news archival data around the Taliban takeover of Afghanistan in August 2021, I establish that the large refugee movements triggered by this event led to a substantial increase in the political salience of immigration across Europe. 3/8
May 21, 2025 at 10:03 AM
Analyzing social media and news archival data around the Taliban takeover of Afghanistan in August 2021, I establish that the large refugee movements triggered by this event led to a substantial increase in the political salience of immigration across Europe. 3/8
🇪🇺🇦🇫 Published Today in CPS 🇪🇺🇦🇫
“Stand by those who share our values” – how refugees fleeing the Taliban improved European attitudes toward immigration
Article: journals.sagepub.com/doi/10.1177/...
Pre-print: osf.io/preprints/os...
Thread: 1/8
“Stand by those who share our values” – how refugees fleeing the Taliban improved European attitudes toward immigration
Article: journals.sagepub.com/doi/10.1177/...
Pre-print: osf.io/preprints/os...
Thread: 1/8
May 21, 2025 at 10:03 AM
🇪🇺🇦🇫 Published Today in CPS 🇪🇺🇦🇫
“Stand by those who share our values” – how refugees fleeing the Taliban improved European attitudes toward immigration
Article: journals.sagepub.com/doi/10.1177/...
Pre-print: osf.io/preprints/os...
Thread: 1/8
“Stand by those who share our values” – how refugees fleeing the Taliban improved European attitudes toward immigration
Article: journals.sagepub.com/doi/10.1177/...
Pre-print: osf.io/preprints/os...
Thread: 1/8
A shipwreck with media coverage (based on news-archival data) comparable to Lampedusa 2013 (red line) is predicted to have less than 1/10th of the originally estimated effect. A shipwreck with twice that level of media coverage is predicted to have roughly 1/5th of the original effect. 13/15
March 20, 2025 at 10:24 AM
A shipwreck with media coverage (based on news-archival data) comparable to Lampedusa 2013 (red line) is predicted to have less than 1/10th of the originally estimated effect. A shipwreck with twice that level of media coverage is predicted to have roughly 1/5th of the original effect. 13/15
Even when taking the original effect for granted though, that one was extremely short-lived. The post-Lampedusa trend is negative (p<0.05) and back-of-the-envelope calculations suggest that the Lampedusa effect vanished after less than two weeks. 12/15
March 20, 2025 at 10:24 AM
Even when taking the original effect for granted though, that one was extremely short-lived. The post-Lampedusa trend is negative (p<0.05) and back-of-the-envelope calculations suggest that the Lampedusa effect vanished after less than two weeks. 12/15
Furthermore, the original effect is also the one with the most uncertainty attached to it. The original study had less than 10% statistical power to detect a small effect. My lowest-powered replication had 87% power to detect a small effect and 99.9% to detect a medium-sized effect. 11/15
March 20, 2025 at 10:24 AM
Furthermore, the original effect is also the one with the most uncertainty attached to it. The original study had less than 10% statistical power to detect a small effect. My lowest-powered replication had 87% power to detect a small effect and 99.9% to detect a medium-sized effect. 11/15
Not only is this by far the largest effect; out of all the other effects, only one even reaches conventional statistical significance. Looking at the distribution of p-values from all effects, those follow almost precisely the uniform distribution you would expect from a true null effect. 10/15
March 20, 2025 at 10:24 AM
Not only is this by far the largest effect; out of all the other effects, only one even reaches conventional statistical significance. Looking at the distribution of p-values from all effects, those follow almost precisely the uniform distribution you would expect from a true null effect. 10/15
But back to the empirics: analyzing regression discontinuities around ESS (wave 6-10) responses before and after 15 of the most lethal shipwreck in the past decade, I show that the effect of the original Lampedusa shipwreck (far left) is a clear outlier, both in Italy and the rest of Europe. 9/15
March 20, 2025 at 10:24 AM
But back to the empirics: analyzing regression discontinuities around ESS (wave 6-10) responses before and after 15 of the most lethal shipwreck in the past decade, I show that the effect of the original Lampedusa shipwreck (far left) is a clear outlier, both in Italy and the rest of Europe. 9/15
While this is mainly an empirical paper, I am very passionate about improving the reliability of the UESD and I hope that my detailed outline of the MUESD workflow can aid others when analyzing frequently occurring events (check especially Appendix A9 for many more details on this). 8/15
March 20, 2025 at 10:24 AM
While this is mainly an empirical paper, I am very passionate about improving the reliability of the UESD and I hope that my detailed outline of the MUESD workflow can aid others when analyzing frequently occurring events (check especially Appendix A9 for many more details on this). 8/15