The Unjournal (Unjournal.org)
@unjournal.bsky.social
Researchers, practitioners, & open science advocates building a better system for research evaluation. Nonprofit. We commission public evaluation & rating of hosted work. To make rigorous research more impactful, & impactful research more rigorous.
How long do Unjournal evaluators take to do their evaluations? For our academic stream, about 8 hours on average, with substantial dispersion.
(Scatterplot: imputed data from 40 quantifiable responses, not including applied stream nor those who used the PubPub form.)
(Scatterplot: imputed data from 40 quantifiable responses, not including applied stream nor those who used the PubPub form.)
November 12, 2025 at 11:48 PM
How long do Unjournal evaluators take to do their evaluations? For our academic stream, about 8 hours on average, with substantial dispersion.
(Scatterplot: imputed data from 40 quantifiable responses, not including applied stream nor those who used the PubPub form.)
(Scatterplot: imputed data from 40 quantifiable responses, not including applied stream nor those who used the PubPub form.)
If you're among the 2000+ authors citing research evaluated by Unjournal.org, go to unjournal.pubpub.org to learn how commissioned experts rated & assessed that research.
Top 'citers of unjournal-evaluated research': https://bit.ly/3WRHc8W include Esther Duflo, Julian Jamison, & Berk Özler
Top 'citers of unjournal-evaluated research': https://bit.ly/3WRHc8W include Esther Duflo, Julian Jamison, & Berk Özler
October 22, 2025 at 4:16 PM
If you're among the 2000+ authors citing research evaluated by Unjournal.org, go to unjournal.pubpub.org to learn how commissioned experts rated & assessed that research.
Top 'citers of unjournal-evaluated research': https://bit.ly/3WRHc8W include Esther Duflo, Julian Jamison, & Berk Özler
Top 'citers of unjournal-evaluated research': https://bit.ly/3WRHc8W include Esther Duflo, Julian Jamison, & Berk Özler
Does "Irrigation strengthen climate resilience"? -- Unjournal evaluation
https://bit.ly/3J1iMqk
BenYishay et al '24: River-based irrigation boosted #agricultural output & child #nutrition & reduced conflict in supported communities, but had negative spillovers on surrounding areas.
--->
https://bit.ly/3J1iMqk
BenYishay et al '24: River-based irrigation boosted #agricultural output & child #nutrition & reduced conflict in supported communities, but had negative spillovers on surrounding areas.
--->
October 8, 2025 at 9:34 PM
Does "Irrigation strengthen climate resilience"? -- Unjournal evaluation
https://bit.ly/3J1iMqk
BenYishay et al '24: River-based irrigation boosted #agricultural output & child #nutrition & reduced conflict in supported communities, but had negative spillovers on surrounding areas.
--->
https://bit.ly/3J1iMqk
BenYishay et al '24: River-based irrigation boosted #agricultural output & child #nutrition & reduced conflict in supported communities, but had negative spillovers on surrounding areas.
--->
I love what #SciPost does & their narrative.
Much agrees w/ Unjournal.org practices.
Some differences (Us/Them)
1. Quant. ratings vs accept/reject
2. Publish all evaluations vs. only if ‘accepted’
3. Fields & applied focus
4. We pay evaluators
5. We don’t 'publish' papers, just evaluations
Much agrees w/ Unjournal.org practices.
Some differences (Us/Them)
1. Quant. ratings vs accept/reject
2. Publish all evaluations vs. only if ‘accepted’
3. Fields & applied focus
4. We pay evaluators
5. We don’t 'publish' papers, just evaluations
October 3, 2025 at 6:25 PM
I love what #SciPost does & their narrative.
Much agrees w/ Unjournal.org practices.
Some differences (Us/Them)
1. Quant. ratings vs accept/reject
2. Publish all evaluations vs. only if ‘accepted’
3. Fields & applied focus
4. We pay evaluators
5. We don’t 'publish' papers, just evaluations
Much agrees w/ Unjournal.org practices.
Some differences (Us/Them)
1. Quant. ratings vs accept/reject
2. Publish all evaluations vs. only if ‘accepted’
3. Fields & applied focus
4. We pay evaluators
5. We don’t 'publish' papers, just evaluations
You also raise an important question:
October 1, 2025 at 6:58 PM
You also raise an important question:
Climate change, environmental econ., biodiversity: Of the 50+ papers Unjournal.org commissioned for evaluation and assessment, ten focus on these areas, offering insight into methods & policy.
See our 'collection': https://bit.ly/4pKF0NT -->
See our 'collection': https://bit.ly/4pKF0NT -->
September 29, 2025 at 11:34 AM
Climate change, environmental econ., biodiversity: Of the 50+ papers Unjournal.org commissioned for evaluation and assessment, ten focus on these areas, offering insight into methods & policy.
See our 'collection': https://bit.ly/4pKF0NT -->
See our 'collection': https://bit.ly/4pKF0NT -->
Unjournal.pubpub.org: 50+ detailed packages evaluating research w/ potential for impact.
But who funded the *research*? GiveWell @poverty-action.bsky.social, @gatesfoundation.bsky.social, ERC, NIH, @wellcometrust.bsky.social, & ~40 others
See our dashboard: tinyurl.com/22k6ashr
-->
But who funded the *research*? GiveWell @poverty-action.bsky.social, @gatesfoundation.bsky.social, ERC, NIH, @wellcometrust.bsky.social, & ~40 others
See our dashboard: tinyurl.com/22k6ashr
-->
September 27, 2025 at 11:52 AM
Unjournal.pubpub.org: 50+ detailed packages evaluating research w/ potential for impact.
But who funded the *research*? GiveWell @poverty-action.bsky.social, @gatesfoundation.bsky.social, ERC, NIH, @wellcometrust.bsky.social, & ~40 others
See our dashboard: tinyurl.com/22k6ashr
-->
But who funded the *research*? GiveWell @poverty-action.bsky.social, @gatesfoundation.bsky.social, ERC, NIH, @wellcometrust.bsky.social, & ~40 others
See our dashboard: tinyurl.com/22k6ashr
-->
So measuring moral preferences is hard, context matters enormously. And maybe we shouldn't assume people have neat, stable moral philosophies.
Full evaluations at tinyurl.com/4hk8mr95 and links within.
What do you think about the author's approach?
#econsky #moralphilosophy #psychology
Full evaluations at tinyurl.com/4hk8mr95 and links within.
What do you think about the author's approach?
#econsky #moralphilosophy #psychology
September 24, 2025 at 3:39 PM
So measuring moral preferences is hard, context matters enormously. And maybe we shouldn't assume people have neat, stable moral philosophies.
Full evaluations at tinyurl.com/4hk8mr95 and links within.
What do you think about the author's approach?
#econsky #moralphilosophy #psychology
Full evaluations at tinyurl.com/4hk8mr95 and links within.
What do you think about the author's approach?
#econsky #moralphilosophy #psychology
Experiments: social preferences games & social dillemmas.
Standard student participant pool
Authors "no evidence of stable individual preference types across situations."
So people who refuse to "pull the lever to switch tracks" might happily tell a white lie for charity?
Standard student participant pool
Authors "no evidence of stable individual preference types across situations."
So people who refuse to "pull the lever to switch tracks" might happily tell a white lie for charity?
September 24, 2025 at 3:38 PM
Experiments: social preferences games & social dillemmas.
Standard student participant pool
Authors "no evidence of stable individual preference types across situations."
So people who refuse to "pull the lever to switch tracks" might happily tell a white lie for charity?
Standard student participant pool
Authors "no evidence of stable individual preference types across situations."
So people who refuse to "pull the lever to switch tracks" might happily tell a white lie for charity?
Did you know The Unjournal has a researcher and evaluator pool with:
309 prioritized members, 170 highly prioritized (rated 65%+)
110 with PhD degrees
147 Economists
Dashboard: https://bit.ly/4gxFZws
How to get involved & use our work: https://bit.ly/3ImqTxu
309 prioritized members, 170 highly prioritized (rated 65%+)
110 with PhD degrees
147 Economists
Dashboard: https://bit.ly/4gxFZws
How to get involved & use our work: https://bit.ly/3ImqTxu
September 20, 2025 at 12:45 PM
Did you know The Unjournal has a researcher and evaluator pool with:
309 prioritized members, 170 highly prioritized (rated 65%+)
110 with PhD degrees
147 Economists
Dashboard: https://bit.ly/4gxFZws
How to get involved & use our work: https://bit.ly/3ImqTxu
309 prioritized members, 170 highly prioritized (rated 65%+)
110 with PhD degrees
147 Economists
Dashboard: https://bit.ly/4gxFZws
How to get involved & use our work: https://bit.ly/3ImqTxu
13/52 of these have been published in peer reviewed journals (before, after, or during our evaluations).
September 7, 2025 at 5:54 PM
13/52 of these have been published in peer reviewed journals (before, after, or during our evaluations).
Yep. Unjournal ratings are on a continous scale, with overall ratings as well as multiple dimensions.
August 13, 2025 at 5:36 PM
Yep. Unjournal ratings are on a continous scale, with overall ratings as well as multiple dimensions.
How do you feel about ratings (like those we offer at unjournal.org)?
August 13, 2025 at 5:33 PM
How do you feel about ratings (like those we offer at unjournal.org)?
Fwiw their approach also provides closer links to “publication"
August 13, 2025 at 3:17 PM
Fwiw their approach also provides closer links to “publication"
You can view these and other commended evaluations here: bit.ly/3Je5eY9
You can also find out more details about our small incentives draw on our public Coda page: bit.ly/small_incentives
You can also find out more details about our small incentives draw on our public Coda page: bit.ly/small_incentives
August 12, 2025 at 4:42 PM
You can view these and other commended evaluations here: bit.ly/3Je5eY9
You can also find out more details about our small incentives draw on our public Coda page: bit.ly/small_incentives
You can also find out more details about our small incentives draw on our public Coda page: bit.ly/small_incentives
Will we soon eat meat produced in a large bioreactor rather than on factory farms?
Should animal welfare advocates fund this development?
Unjournal's first "Pivotal Question" considers this:
https://bit.ly/4m5ontN
https://bit.ly/4nGD8EH
Should animal welfare advocates fund this development?
Unjournal's first "Pivotal Question" considers this:
https://bit.ly/4m5ontN
https://bit.ly/4nGD8EH
July 9, 2025 at 7:07 PM
Will we soon eat meat produced in a large bioreactor rather than on factory farms?
Should animal welfare advocates fund this development?
Unjournal's first "Pivotal Question" considers this:
https://bit.ly/4m5ontN
https://bit.ly/4nGD8EH
Should animal welfare advocates fund this development?
Unjournal's first "Pivotal Question" considers this:
https://bit.ly/4m5ontN
https://bit.ly/4nGD8EH
Drug prices too LOW?
Frech et al, 2023's model & evidence suggest even US prices (highest in OECD) don't adequately incentivize R&D & USA should advocate higher global prices.
2 Unjournal evaluators: strong doubts & constructive criticism: https://bit.ly/4l8BCK7
Frech et al, 2023's model & evidence suggest even US prices (highest in OECD) don't adequately incentivize R&D & USA should advocate higher global prices.
2 Unjournal evaluators: strong doubts & constructive criticism: https://bit.ly/4l8BCK7
June 30, 2025 at 10:56 PM
Drug prices too LOW?
Frech et al, 2023's model & evidence suggest even US prices (highest in OECD) don't adequately incentivize R&D & USA should advocate higher global prices.
2 Unjournal evaluators: strong doubts & constructive criticism: https://bit.ly/4l8BCK7
Frech et al, 2023's model & evidence suggest even US prices (highest in OECD) don't adequately incentivize R&D & USA should advocate higher global prices.
2 Unjournal evaluators: strong doubts & constructive criticism: https://bit.ly/4l8BCK7
Unjournal makes all evaluations public & allows evaluators to choose anonymity or not.
I expected those who opt into anonymity to be more negative. Plotted overall ratings --> mixed bag.
https://bit.ly/4kaFQzq
I expected those who opt into anonymity to be more negative. Plotted overall ratings --> mixed bag.
https://bit.ly/4kaFQzq
June 27, 2025 at 4:29 PM
Unjournal makes all evaluations public & allows evaluators to choose anonymity or not.
I expected those who opt into anonymity to be more negative. Plotted overall ratings --> mixed bag.
https://bit.ly/4kaFQzq
I expected those who opt into anonymity to be more negative. Plotted overall ratings --> mixed bag.
https://bit.ly/4kaFQzq
Zimbabwe grandmas giving 45 minutes of therapy– among the most effective ways to boost wellbeing, says HLI meta-analysis/review/CEA.
bit.ly/44cvnh0 Both evaluators had ~confidence in main results. Also critiques & suggestions; authors responded in detail. ->
bit.ly/44cvnh0 Both evaluators had ~confidence in main results. Also critiques & suggestions; authors responded in detail. ->
June 20, 2025 at 8:43 PM
Zimbabwe grandmas giving 45 minutes of therapy– among the most effective ways to boost wellbeing, says HLI meta-analysis/review/CEA.
bit.ly/44cvnh0 Both evaluators had ~confidence in main results. Also critiques & suggestions; authors responded in detail. ->
bit.ly/44cvnh0 Both evaluators had ~confidence in main results. Also critiques & suggestions; authors responded in detail. ->
Strong evidence "university patents actually crowds out private patenting"?
Some debate on this, thanks to our evaluators (Ioannis Bournakis and an anonymous one) and the detailed response from the authors. @ashisharora.bsky.social, Belenzon, Cioca, Sheer, and Zhang.
Some debate on this, thanks to our evaluators (Ioannis Bournakis and an anonymous one) and the detailed response from the authors. @ashisharora.bsky.social, Belenzon, Cioca, Sheer, and Zhang.
June 19, 2025 at 8:47 PM
Strong evidence "university patents actually crowds out private patenting"?
Some debate on this, thanks to our evaluators (Ioannis Bournakis and an anonymous one) and the detailed response from the authors. @ashisharora.bsky.social, Belenzon, Cioca, Sheer, and Zhang.
Some debate on this, thanks to our evaluators (Ioannis Bournakis and an anonymous one) and the detailed response from the authors. @ashisharora.bsky.social, Belenzon, Cioca, Sheer, and Zhang.
"Can we trust social science yet?" @ryancbriggs.net asks in @asterisk_mag.
https://bit.ly/4kYyHTx
His verdict: "at present... usually 'no.'"
But he sees some points of light, including unjournal.org & i4replication.org (he's on both teams) -->
https://bit.ly/4kYyHTx
His verdict: "at present... usually 'no.'"
But he sees some points of light, including unjournal.org & i4replication.org (he's on both teams) -->
June 11, 2025 at 2:45 PM
"Can we trust social science yet?" @ryancbriggs.net asks in @asterisk_mag.
https://bit.ly/4kYyHTx
His verdict: "at present... usually 'no.'"
But he sees some points of light, including unjournal.org & i4replication.org (he's on both teams) -->
https://bit.ly/4kYyHTx
His verdict: "at present... usually 'no.'"
But he sees some points of light, including unjournal.org & i4replication.org (he's on both teams) -->
Cannon Cloud updated their ratings in light of discussions with the anonymous evaluator. See:
unjournal.pubpub.org/pub/evalsumn...
unjournal.pubpub.org/pub/evalsumn...
June 6, 2025 at 1:32 PM
Cannon Cloud updated their ratings in light of discussions with the anonymous evaluator. See:
unjournal.pubpub.org/pub/evalsumn...
unjournal.pubpub.org/pub/evalsumn...
Sampling/statement selection bias concerns:
E1: "... whether the sample of respondents is disproportionately drawn from individuals already aligned with AGI safety priorities"
E2: ..."where statements selected from labs practices are agreed on by labs themselves"
E1: "... whether the sample of respondents is disproportionately drawn from individuals already aligned with AGI safety priorities"
E2: ..."where statements selected from labs practices are agreed on by labs themselves"
June 3, 2025 at 11:08 AM
Sampling/statement selection bias concerns:
E1: "... whether the sample of respondents is disproportionately drawn from individuals already aligned with AGI safety priorities"
E2: ..."where statements selected from labs practices are agreed on by labs themselves"
E1: "... whether the sample of respondents is disproportionately drawn from individuals already aligned with AGI safety priorities"
E2: ..."where statements selected from labs practices are agreed on by labs themselves"
For other packages, evaluators responded specifically to this ~'most important robustness checks' question.
We're curating these (for replicators, students, future research, etc.) here: coda.io/@contact-unj...
More coming soon.
We're curating these (for replicators, students, future research, etc.) here: coda.io/@contact-unj...
More coming soon.
May 9, 2025 at 2:37 PM
For other packages, evaluators responded specifically to this ~'most important robustness checks' question.
We're curating these (for replicators, students, future research, etc.) here: coda.io/@contact-unj...
More coming soon.
We're curating these (for replicators, students, future research, etc.) here: coda.io/@contact-unj...
More coming soon.
We ask evaluators to identify and assess the "paper's main claim". They identified similar claims, but disagreed about the credibility of these claims.
#openscience
#openscience
May 9, 2025 at 1:25 PM
We ask evaluators to identify and assess the "paper's main claim". They identified similar claims, but disagreed about the credibility of these claims.
#openscience
#openscience