Jamie Cummins
LightNews LightNews
jamiecummins.bsky.social
Jamie Cummins
@jamiecummins.bsky.social
2.8K followers 690 following 910 posts
Currently a visiting researcher at @bennettoxford.bsky.social. Normally at Uni of Bern.
Meta-scientist building tools to help other scientists. NLP, simulation, & LLMs.
Creator and developer of RegCheck (https://regcheck.app).
1/4 of @error.reviews.
🇮🇪
Posts Replies Media Videos
Pinned
jamiecummins.bsky.social
Jamie Cummins @jamiecummins.bsky.social · Jul 23
Introducing RegCheck: a tool which uses Large Language Models to automatically compare preregistered protocols with their corresponding published papers and highlights deviations.

@malte.the100.ci @ianhussey.bsky.social @ruben.the100.ci @bjoernhommel.bsky.social

regcheck.app
RegCheck.app
RegCheck is an AI tool to compare preregistrations with papers instantly.
regcheck.app
Reposted by Jamie Cummins
deevybee.bsky.social
so maybe @mdpiopenaccess.bsky.social could make a resolution to remove all editors and reviewers who have accepted nonsensical papers. It would be quite a cull
mdpiopenaccess.bsky.social MDPIOpenAccess @mdpiopenaccess.bsky.social · 5d
As a pioneer in Open Access scholarly publishing, MDPI upholds high ethical publishing standards across its journals.

Learn more about how MDPI continues to strengthen its publication ethics policies: buff.ly/nssdnLb

#MDPI #PublicationEthics #OpenAccess
January 2, 2026 at 7:29 AM
Reposted by Jamie Cummins
dermotlynott.bsky.social
Journal article: “the data are available on request”

The data:
Redacted Epstein files Redacted Epstein files. Basically completely blank Redacted Epstein files. Basically completely blank Redacted Epstein files. Basically completely blank
December 20, 2025 at 11:33 AM
Reposted by Jamie Cummins
malte.the100.ci
I regularly cite Prinz et al.(www.nature.com/articles/nrd...) as a ref for low replicability in (non-psych) preclinical research. Believe it or not: They've omitted which studies they attempted to replicate!

I'm guessing this isn't news to everyone, but it was to me. Bizarre.
Believe it or not: how much can we rely on published data on potential drug targets? - Nature Reviews Drug Discovery
Nature Reviews Drug Discovery - Believe it or not: how much can we rely on published data on potential drug targets?
www.nature.com
December 19, 2025 at 1:19 PM
Reposted by Jamie Cummins
debruine.bsky.social
The package formerly known as papercheck has changed its name to metacheck! We're checking more than just papers, with functions to assess OSF projects, github repos, and AsPredicted pre-registrations, with more being developed all the time.

scienceverse.github.io/metacheck/
Check Research Outputs for Best Practices
A modular, extendable system for automatically checking research outputs for best practices using text search, R code, and/or (optional) LLM queries.
scienceverse.github.io
November 3, 2025 at 4:20 PM
Reposted by Jamie Cummins
ruben.the100.ci
Finally, @bjoernhommel.bsky.social's and my paper introducing the SurveyBot3000 is officially out in AMPPS. It's a fine-tuned language model that guesstimates correlations between survey items from text alone. Not perfectly, but useful for search, for example.
journals.sagepub.com/doi/10.1177/...
Fig. 3. Scatter plots of the synthetic and empirical estimates, validation study (Stage 2). Showing N = 30,135 item-pair correlations, N = 257 scale reliabilities, and N = 1,568 scale-pair correlations for (top) the pretrained SBERT model and (bottom) the fine-tuned SurveyBot3000 model. SBERT = all-mpnet-base-v2 model. Fig. 4. Prediction error of the synthetic estimates, validation study (Stage 2). Our prediction model allowed the error term to vary freely according to the predictor, the synthetic estimate. The thin-plate splines show that some synthetic estimates were predictably more accurate. Fig. 5. Accuracy by domain. Accuracy differed across domains. SurveyBot3000 accuracy (colored) was always higher than SBERT accuracy (gray). Results were largely consistent whether accuracy of items was tested (left, circle) within domains or (right, cross) across domains. Fig. 1. Multistep training procedure for the SurveyBot3000, which produces synthetic estimates of interitem correlations. (a) Pretraining base model (SBERT). (b) Fine-tuning SurveyBot3000. (c) Validation. SBERT = all-mpnet-base-v2 model.
December 18, 2025 at 8:21 PM
jamiecummins.bsky.social
Contrasting to silicon samples, this type of method based on text embeddings seems to be both more principled and more robust in making predictions about scale and item correlations. Super happy to see this great work out!
ruben.the100.ci Ruben C. Arslan @ruben.the100.ci · 18d
Finally, @bjoernhommel.bsky.social's and my paper introducing the SurveyBot3000 is officially out in AMPPS. It's a fine-tuned language model that guesstimates correlations between survey items from text alone. Not perfectly, but useful for search, for example.
journals.sagepub.com/doi/10.1177/...
Fig. 3. Scatter plots of the synthetic and empirical estimates, validation study (Stage 2). Showing N = 30,135 item-pair correlations, N = 257 scale reliabilities, and N = 1,568 scale-pair correlations for (top) the pretrained SBERT model and (bottom) the fine-tuned SurveyBot3000 model. SBERT = all-mpnet-base-v2 model. Fig. 4. Prediction error of the synthetic estimates, validation study (Stage 2). Our prediction model allowed the error term to vary freely according to the predictor, the synthetic estimate. The thin-plate splines show that some synthetic estimates were predictably more accurate. Fig. 5. Accuracy by domain. Accuracy differed across domains. SurveyBot3000 accuracy (colored) was always higher than SBERT accuracy (gray). Results were largely consistent whether accuracy of items was tested (left, circle) within domains or (right, cross) across domains. Fig. 1. Multistep training procedure for the SurveyBot3000, which produces synthetic estimates of interitem correlations. (a) Pretraining base model (SBERT). (b) Fine-tuning SurveyBot3000. (c) Validation. SBERT = all-mpnet-base-v2 model.
December 18, 2025 at 8:33 PM
jamiecummins.bsky.social
Great thread about a very timely preprint. Couldn’t agree more with this point especially. When results from LLMs in “silicon samples” can vary substantially as a function of many analytic decisions, systematically making decisions erases much perceived efficiency.
arxiv.org/abs/2509.133...
December 18, 2025 at 6:16 PM
Reposted by Jamie Cummins
jessicahullman.bsky.social
Many think LLM-simulated participants can transform behavioral science. But there's been a lack of accessible discussion of what it means to validate LLMs for behavioral scientists. Under what conditions can we trust LLMs to learn about human parameters? Our paper maps the validation landscape.
1/
December 18, 2025 at 5:53 PM
Reposted by Jamie Cummins
error.reviews
At ERROR, we cannot compete with million-dollar bounties for whistleblowers. But it is great to see sleuthing work rewarded, and institutions admitting when their researchers engaged in misconduct.
December 18, 2025 at 12:31 PM
Reposted by Jamie Cummins
malte.the100.ci
While we are unable to offer publication in Nature Aging, we believe your manuscript may be suitable for our Gold Open Access sister journal.
December 17, 2025 at 7:30 PM
Reposted by Jamie Cummins
error.reviews
Post-publication peer review is at it best when it's thoughtful, scrupulous, steeped in detail – and challenges key claims of the paper. @janhove.bsky.social's discussion of a recent paper on multilingualism exemplifies this.
janhove.bsky.social Jan Vanhove @janhove.bsky.social · 21d
A recent study purports to have found that multilingualism protects against accelerated ageing. I've taken a closer look at it, and it doesn't look good.

New blog post: "Does multilingualism really protect against accelerated ageing? Some critical comments"
janhove.github.io/posts/2025-1...
Positive trend between monolingualism and accelerated ageing. Negative trend between per capita GDP and accelerated ageing.
December 17, 2025 at 10:01 AM
Reposted by Jamie Cummins
sabrinanorwood.bsky.social
Did you know that from tomorrow, Qualtrics is offering synthetic panels (AI-generated participants)?

Follow me down a rabbit hole I'm calling "doing science is tough and I'm so busy, can't we just make up participants?"
Text reads: About synthetic panels Recruiting the right participants for a study can be difficult. You may not get the exact demographics you need, and the shorter the deadline, the less sure you can be that everyone will answer on time. One possible solution can be to use synthetic panels. Synthetic panels are powered by a first party proprietary AI model developed here at Qualtrics. Our synthetic panel is trained on thousands of responses from a variety of demographic backgrounds in order to more accurately predict how certain populations would respond to a survey. Our synthetic panel is based on the United States General Population, and is only available in English. This panel comes with ready-made quotas and target breakouts in order to represent your chosen population and make it easy to launch your survey right away. Text reads: Question-writing best practices To get the most reliable and actionable results from synthetic audiences, consider these question-writing best practices: Ask forward-looking and attitudinal questions. Synthetic panels perform best with perceptions, preferences, and intent-based questions. For example, “How likely are you to try…?” Synthetic panels are less applicable for studies on past behaviors, detailed recall, brand recall, or awareness questions. For example, “When did you last visit…?” Text reads: Discussion The current study aimed to conduct a meta-analysis of the TPB when applied to health behaviours which addressed the limitations of previous reviews by including only prospective tests of behaviour, applying RE meta-analytic procedures, correcting correlations for sampling and measurement error, and hierarchically analysing the effect of behaviour type and sample and methodological moderators. Some 237 tests were identified which examined relations amongst model components. Overall the analysis indicated that the TPB could explain 19.3% of the variance in behaviour and 44.3% of the variance in intention across studies. This level of prediction of behaviour is slightly lower than that of previous meta-analytic reviews which have found between 27% (Armitage & Conner, 2001; Hagger et al., 2002) and 36% (Trafimow et al., 2002) of the variance in behaviour to be explained by intention and PBC.
December 16, 2025 at 5:38 PM
Reposted by Jamie Cummins
sabrinanorwood.bsky.social
See @jamiecummins.bsky.social 's work
AI-generated ‘participants’ can lead social science experiments astray, study finds
Data produced by “silicon samples” depends on researchers’ exact choice of models, prompts, and settings
www.science.org
December 16, 2025 at 5:38 PM
Reposted by Jamie Cummins
jntowse.bsky.social
Thanks for sharing!

You may be interested in this, which resonates, from @jamiecummins.bsky.social, on the fragility of silicon samples

doi.org/10.48550/arX...
The threat of analytic flexibility in using large language models to simulate human data: A call to attention
Social scientists are now using large language models to create "silicon samples" - synthetic datasets intended to stand in for human respondents, aimed at revolutionising human subjects research. How...
doi.org
December 16, 2025 at 11:43 AM
Reposted by Jamie Cummins
ianhussey.mmmdata.io
A new PNAS paper finds that polarization increased immediately after the release of Lady Gaga’s “Just Dance” and the advent of the late-2000s electro-pop era, which both appeared around the same year, 2008.
jayvanbavel.bsky.social Jay Van Bavel, PhD @jayvanbavel.bsky.social · 22d
A new PNAS paper finds that polarization increased immediately after the invention of smartphones and the advent of social media, which both appeared around the same year, 2008.
www.pnas.org/doi/10.1073/...
December 15, 2025 at 8:21 AM
Reposted by Jamie Cummins
ianhussey.mmmdata.io
Critique argues post-publication peer review "often focuses on minor details" and "risks damaging trust in both the research itself and the peer review process" found to contain:

- Hallucinated references
- Undisclosed COIs (EiC is author)
- 8/9 authors have retractions or related scandals
An expert criticism on post-publication peer review platforms: the case of pubpeer - DARU Journal of Pharmaceutical Sciences
While traditional peer review offers advantages in academic publishing, it is often hampered by significant weaknesses, leading to frustration among many authors. Scientific discoveries after publicat...
link.springer.com
December 13, 2025 at 12:12 PM
Reposted by Jamie Cummins
simine.com
Congratulations to @bethclarke.bsky.social on her PhD graduation!! It’s been amazing watching you become the incredible scholar that you are, thanks for letting me come along for the ride!
You’re the best! 🎉🥂🌟
Simine and Beth in graduation regalia outside Selfie of Beth and Simine holding glasses of sparkling wine in graduation regalia.
December 13, 2025 at 10:55 AM
Reposted by Jamie Cummins
ianhussey.mmmdata.io
Keeping this at hand in case I need to point to it and tap
December 12, 2025 at 4:39 PM
Reposted by Jamie Cummins
ianhussey.mmmdata.io
Many psychologists are treating LLMs as if they are the mind of god.

This study had chatGPT rate how central academic disciplines are to various constructs.

Why would chatGPT know this?

Where is the evidence its ratings are reliable or valid?

compass.onlinelibrary.wiley.com/doi/full/10....
December 10, 2025 at 10:09 AM
jamiecummins.bsky.social
As someone extremely non-aesthetically oriented, I needed this
scientificdiscovery.dev Saloni @scientificdiscovery.dev · 27d
Big new blogpost!

My guide to data visualization, which includes a very long table of contents, tons of charts, and more.

--> Why data visualization matters and how to make charts more effective, clear, transparent, and sometimes, beautiful.
www.scientificdiscovery.dev/p/salonis-gu...
screenshot of my post
December 10, 2025 at 8:39 AM
Reposted by Jamie Cummins
scientificdiscovery.dev
Big new blogpost!

My guide to data visualization, which includes a very long table of contents, tons of charts, and more.

--> Why data visualization matters and how to make charts more effective, clear, transparent, and sometimes, beautiful.
www.scientificdiscovery.dev/p/salonis-gu...
screenshot of my post
December 9, 2025 at 8:28 PM
Reposted by Jamie Cummins
bennettoxford.bsky.social
Tomorrow: sold-out Bennett Symposium at Jesus College www.bennett.ox.ac.uk/events/2025-.... Day 1 = presentations and panels. Day 2 = collaborative sessions and drop-ins with OpenPrescribing & OpenSAFELY teams. If you're attending, see you there 📊
2025 Bennett Institute Medicines Symposium | Bennett Institute for Applied Data Science
The two-day 2025 Bennett Institute Medicines Symposium will take place at Jesus College in Oxford on 10–11 December 2025.
www.bennett.ox.ac.uk
December 9, 2025 at 3:43 PM
Reposted by Jamie Cummins
ianhussey.mmmdata.io
Project Implicit gives you feedback when you complete an IAT, and the IAT literature has aspired to individual level measurement since its inception.

In this article, we quantify the individual level uncertainty around these estimates and show that IAT effects can say very little about individuals.
December 9, 2025 at 12:11 PM
Reposted by Jamie Cummins
mzloteanu.bsky.social
More research like this on other psychometric scales is needed (although the IAT should just be abandoned regardless).

Researchers and practitioners should know the precision of their tools. If someone is 1pt over a cutoff how reliable is this for classification and treatment?
December 9, 2025 at 12:18 PM
jamiecummins.bsky.social
Just published in Behavior Research Methods:

The individual-level precision of implicit measures

w/ @ianhussey.mmmdata.io

🧵👇

link.springer.com/article/10.3...
The individual-level precision of implicit measures - Behavior Research Methods
Implicit measures are used extensively in psychological science. One fundamental goal of these measures is to provide information diagnostic of an individual’s attitudes or beliefs. After 25 years of ...
link.springer.com
December 9, 2025 at 11:20 AM