5/🧵 Relational analysis also identified a role of the dorsal striatum. Despite not being typically seen as an area for semantic coding, striatal regions robustly represent relational content, and the strength of this representation predicts participants’ judgments about item pairs

4/🧵 We used LLM embeddings to model neural representations via fMRI and RSA, which revealed dissociations in information processing. Occipitotemporal structures parse concepts but not relations. Conversely, frontoparietal regions (especially the PFC) almost exclusively encode relational information

3/🧵 Next, we tested relational information. LLM activity for texts like “A prison and poker table” robustly predicts human ratings on the likelihood of finding a poker table in a prison. Further analyses show how LLMs parsing such texts also capture precise propositional relational features

2/🧵 To produce concept embeddings, we submit a short text (“A porcupine”) to contemporary LLMs and extract the LLMs’ residual stream. Leveraging data from our normative feature study, we find that LLM embeddings better predict human-reported propositional features than older models (word2vec, BERT)

🚨 New preprint 🚨

Prior work has mapped how the brain encodes concepts: If you see fire and smoke, your brain will represent the fire (hot, bright) and smoke (gray, airy). But how do you encode features of the fire-smoke relation? We analyzed fMRI with embeddings extracted from LLMs to find out 🧵

I agree that trying to convert aggregate p-values into a replication rate won't be reliable. Nonetheless, a paper's p-values seem to track something awfully related to replicability. Per Figure 6, fragile p-values neatly identify numerous topics and methods known to produce non-replicable findings

Not sure how informative this would be, but an earlier draft included a subfigure trying to demonstrate that there remain a substantial number of likely problematic papers. Somewhat arbitrarily, the figure showed the percentage of papers where a majority of significant p-values were p > .01

Let's define a study as questionable if it doesn't have 80% power when the sample size as 2.5x. Eyeballing based on playing with G*Power, this means a questionable study is one with <45% power. An effect with 45% power will produce a fragile (.01 < p < .05) p-value about 50% of the time

Thanks for sharing this!

Also, sorry about these issues with the y-axes not rendering. I copied below screenshots of what the site should look like. Maybe a different browser would work? I tested the site on PC, Mac, and iPhone with a few browsers. Maybe toggle light/night mode (top right)?

Thanks and sorry about that! They seem to not be rendering. I pasted a screenshot of what it should look like. I can't reproduce your issue, but maybe switching to dark mode (clicking the little moon/sun button in the top right) will fix it? Maybe a different browser? Should be fine on Mac & PC

Plotting the links between fragile p-values and word usage, we can see how various methodologies and disciplines are linked to strong (green) or weak (red) results.

I made a website, where you can look up your own terms and their associations: pbogdan.com/meganal/

However, fragile p-values are also positively tied to university prestige. Top schools more often publish weak results, which was true both in the past and today. This seems to stem from a focus on topics where replicability progress has been slower (e.g., biological psychology)

Nowadays, papers reporting fragile p-values are less likely published in top journals and receive fewer citations. This was not the case historically, and this shift is a powerful indicator of how the field now better value replicability.

I investigated how often papers' significant (p < .05) results are fragile (.01 ≤ p < .05) p-values. An excess of such p-values suggests low odds of replicability.

From 2004-2024, the rates of fragile p-values have gone down precipitously across every psychology discipline (!)

🚨New paper!🚨

Meta-analysis on 4M p-values across 240k psych articles: How has psychology changed since the replication crisis began? How is replicability linked to citations, impact factor, and university prestige? 🧵

Paper: journals.sagepub.com/doi/10.1177/...

Interactive: pbogdan.com/meganal

Nowadays, papers reporting fragile p-values are less likely published in top journals and receive fewer citations. This was not the case historically, and this shift is a powerful indicator of how the field now better value replicability.

I investigated how often papers report fragile (.01  ≤  p  <  .05) p-values. An excess of such p-values suggests low odds of replicability.

From 2004-2024, the rates of fragile p-values have gone down precipitously across every psychology discipline (!). Sample sizes are also going way up

Nowadays, papers reporting fragile p-values are less likely published in top journals and receive fewer citations. This was not the case historically, and this shift is a powerful indicator of how the field now better values replicability.

I investigated how often papers report fragile (.01  ≤  p  <  .05) p-values. An excess of such p-values suggests low odds of replicability.

From 2004-2024, the rates of fragile p-values have gone down precipitously across every psychology discipline (!). Sample sizes are also going way up