Ruben C. Arslan
LightNews LightNews
banner
ruben.the100.ci
Ruben C. Arslan
@ruben.the100.ci
4.4K followers 720 following 1.4K posts
Bayescurious evidence enthusiast at the100.ci
Topics: evolution, ovulation, mutation, intelligence, personality, sexuality, R, open science & source tools. https://rubenarslan.github.io/
Posts Replies Media Videos
ruben.the100.ci
arxiv.org/pdf/2508.01390
have you seen this?
Box 1 — A Case Study in LLM Pollution Detection and Incidence In a recent study we conducted with participants recruited via Prolific, we found that a lack of bot protection measures led to an alarming rate of seemingly bot-generated answers. This example is used here illustratively to demonstrate the types of challenges that can arise in practice rather than to report new empirical findings. In our initial pilot study, we only tracked copying and pasting on a page containing an openended question. Notably, 45% of participants engaged in one or both of these actions, suggesting potential cases of Partial LLM Mediation and Full LLM Delegation. Even to an untrained eye, some of these responses were obviously generated by LLMs, representing, for instance, cases of overly-abstract summarisations of all previous instructions. Given these results, we implemented the following measures for the main study: • Implemented two reCAPTCHA v2 checkpoints at the beginning and middle of our survey, asking participants to confirm that they are not bots by checking a box and potentially completing a short validation test (0.2% failed to advance); • Activated Qualtrics’ version of reCAPTCHA v3, which uses information about users’ behaviour and past activity to assign each a score between 0 and 1, with lower scores indicating likely bot activity (2.7% participants scored lower than 0.7, a threshold we determined on the basis of our piloting, with the lowest score being 0.2); • Added a honeypot question that included white text invisible to humans, but readable to bots scraping all content (1.6% participants included our study-irrelevant keyword “hazelnut”; out of these 16 participants, only 2 had low reCAPTCHA v3 scores); • Prevented copy-pasting, while still tracking which participants attempted these actions (4.7% participants attempted to copy and/or paste text). While these incidence rates were less concerning than those observed during piloting, we still identified identical nonse…
November 18, 2025 at 9:47 PM
ruben.the100.ci
18s gemini 3 pro. Also, maybe you could do better when you were an eight year old child, not sure this is generally true ;-)
November 18, 2025 at 9:01 PM
ruben.the100.ci
maybe we're talking past each other. with this minimal sim, the pgi-top-ranked is worse than a random pick 30% of the time. is this what you mean? I don't disagree with that. But you have to choose somehow, and random picking clearly does worse more often.
November 13, 2025 at 9:27 PM
ruben.the100.ci
how is this not putting the decision problem on the family level?
November 13, 2025 at 9:10 PM
ruben.the100.ci
Great that you show the raw data, but it makes me wonder. Did you try modelling this as an ordinal outcome? Because the interaction effect looks like it could be fully explained by the item floor in ideation limiting variation at high levels of functioning. journals.sagepub.com/doi/10.1177/...
Simulated data showing a spurious interaction arise due to a ceiling effect.
November 6, 2025 at 9:24 AM
ruben.the100.ci
I built a DAG diagram with garden hoses for teaching.
Pictured: a collider bias diagram, inspired by a blocked pipe situation I experienced (which I credit with giving me the intuition though it also ruined my belongings in the flooded cellar).
me with some garden hoses connected in a X -> Z <- Y fashion. If I shut the valve at Z, water from X spills out at Y
October 28, 2025 at 5:50 PM
ruben.the100.ci
Ok, I gave it my best shot, though of course it depends on the pictures. Chocolates in a bowl wrapped or unwrapped etc.
> library(tidyverse) > > squish <- function(x) { + x <- round(x) + x[x>7] <- 7 + x[x<1] <- 1 + x + } > nonserious <- 0.1 > N <- 500 > dt <- tibble( + serious = sample(c(F, T), N, prob = c(nonserious, 1-nonserious), replace = T), + poop = if_else(serious, 1, sample(1:7, N, T)), + chocolate = if_else(serious, squish(rnorm(N, 6.5, sd = 0.5)), sample(1:7, N, T)), + ) > > table(dt$serious) FALSE TRUE 44 456 > hist(dt$poop) > hist(dt$chocolate) > > effsize::cohen.d(dt$poop, dt$chocolate, paired = T) Cohen's d d estimate: -4.866682 (large) 95 percent confidence interval: lower upper -5.409463 -4.323901
October 13, 2025 at 4:06 PM
ruben.the100.ci
Our fragmentation paper is now finally out! I put some of the dumb quips that didn't make the cut in the alt texts.
journals.sagepub.com/doi/10.1177/...
A fragmented field. If we conceive of the constructs and concepts studied in behavioral science as a map, we would find that it is highly fragmented and directions are hard to come by. Scientists can hardly stand on each others' shoulders if they cannot manage to meet on common ground. Fragmentation has worsened, not decreased, as the field has grown. Partly, this happens because we have too many reverse Columbuses, who, in search of prestige, set out to find a new continent, but just end up renaming India. But partly, we face a real, solvable search problem when trying to connect our fuzzy constructs and flexible measures. Most measures are used only once. To be clear, we do not want to prevent or reduce refinements of existing constructs and measures. Revisions, translations, and other refinements can contribute to a more coherent, organized literature and improve measurement. We are most concerned with the measures conceived with limited planning and released into the literature without much commitment or much of a life expectancy. In ontologies, these are sometimes referred to as “orphan nodes.”
September 29, 2025 at 10:37 AM
ruben.the100.ci
but it might work for us
August 14, 2025 at 8:10 AM
ruben.the100.ci
I made a "bubble" quiz for my class. They got the frequency of veganism and smoking quite wrong, but were fairly good on frequency of "migration background" and "never travelled abroad" even though they were selected on those traits too.
Bubble plot
July 11, 2025 at 6:56 AM
ruben.the100.ci
analysing judgments from this app whoknows.uni-muenster.de cross-classified by item, target and judge
July 10, 2025 at 1:25 PM
ruben.the100.ci
New results from one of the few preregistered, placebo-controlled combined oral contraceptive RCTs. No pill effects found on preferences for masculine or symmetric faces, no menstrual cycle effects.
N=340, so small effects may have gone undetected.
www.sciencedirect.com/science/arti...
We provide the first evidence from a randomized double-blind placebo control study of oral contraceptives on women's preferences for masculine and symmetric faces in men. We find no evidence of an effect. Similar, our correlational results testing for an association between menstrual cycle phases and facial preferences fail to reject the null hypothesis of no association, as do our estimated correlations between hormone levels and facial preferences. Although the early literature largely found support for variation in facial preferences over the menstrual cycle and with oral contraceptive use (Feinberg et al., 2008; Johnston et al., 2001; Little et al., 2002; Little et al., 2013; Penton-Voak et al., 1999; Roney & Simmons, 2008), our null results align with more recent large-sample correlational studies (Jones et al., 2018; Marcinkowska et al., 2019). One possible motivation of the hypothesis that oral contraceptives lower the preferences for facial masculinity and symmetry is in terms of the dual mating theory. In a recent review of the literature Jones et al. (2019) point out several methodological problems in the early work testing this hypothesis, such as low statistical power, and conclude that recent and methodologically stronger work fails to support the hypothesis. Our findings lend further support for their conclusions. There are also several reasons for why false positive results could have been published in the earlier literature. Low statistical power increases the probability for false positive results as well as inflated effect sizes. Many of the older studies also rely on self-reported menstrual cycle data, which is subject to mismeasurement. In addition, without pre-analysis plans there is room for the so-called “researcher degrees of freedom” (Simmons et al., 2011) to influence the analysis such that statistically significant results become more likely to be false positive ones. This can occur through excluding observations, using covariates, testing… Table 2. The effect of OC on facial preferences. Empty Cell % Change in Facial Preferences (Follow-Up - Baseline) Difference P-value of difference Empty Cell Treatment (N = 165) Placebo (N = 168) Empty Cell Empty Cell Masculine Faces −0.083 −0.034 −0.049 0.162 Empty Cell (0.026) (0.023) (0.035) Symmetric Faces −0.025 0.027 −0.052 0.132 Empty Cell (0.024) (0.025) (0.034)
June 23, 2025 at 6:08 AM
ruben.the100.ci
Interestingly, self report estimates are higher than more solidly grounded estimates according to the paper. Now, I don't trust this MDPI paper very much, it certainly has some dodgy plots (negative prevalences? I knew women's issues suffer from low visibility, but that's a bit much).
June 16, 2025 at 9:08 AM
ruben.the100.ci
The absolute numbers go down when we lump revised and translated scales with the original. Abbreviations & changes are only split if some effort went into the abbreviation (i.e. a validated short form). All the ad hoc changes (especially for tasks like CRTT, IAT, IGT) are not catalogued/undercounted
June 13, 2025 at 11:53 AM
ruben.the100.ci
There's a little interactive supplementary website for this paper. You can compare subdisciplines, e.g. I/O psych is more fragmented than educational, perhaps because IQ tests are harder to create ad hoc.
rubenarslan.github.io/construct_pr...
Treemap showing fragmentation across industrial/organizational psychology. Fragmentation (Hill-Shannon Diversity) 𝐷=671. Treemap showing fragmentation across Educational and Developmental Psychology. Fragmentation (Hill-Shannon Diversity) 𝐷=429
June 13, 2025 at 10:50 AM
ruben.the100.ci
Now, 'funnily', using number of coded tests to normalize instead of number of distinct tests was exactly what I did in our first brief comment on the matter — which I then corrected because a) we had incorrectly described it and b) I thought it was more correct.
rubenarslan.github.io/mistakes.htm...
Normalised Shannon entropy. Dotted pre-correction, normalised using number of tests, solid after, normalised using number of distinct tests.
June 13, 2025 at 10:50 AM
ruben.the100.ci
We also ran a rarefaction simulation where we included a 25% of off-by-one errors because or hand recodings of the PsycInfo database showed quite a few such errors. But this doesn't seem to matter much.
June 13, 2025 at 10:50 AM
ruben.the100.ci
There are different efforts to equalize these numbers, but both focus on the number of individuals sampled rather than the number of distinct species (richness, which is biased by effort). We used rarefaction (downsampling) and equalization by asymptotic sample coverage.
measurement fragmentation standardized using fragmentation, with different hill numbers fragmentation standardised using asymptotic coverage
June 13, 2025 at 10:50 AM
ruben.the100.ci
Ecologists now usually use Hill numbers, which are in the metric of number of species. Changes in Hill numbers are also more interpretable, though I wouldn't use the word 'intuitive'.
Here, we report results across years in the Hill-Shannon diversity metric (exponentiated Shannon entropy) using asymptotic coverage standardization. The Hill-Shannon diversity is more interpretable than Shannon entropy because changes in it are linear in the unit of number of measures/constructs. If the Hill-Shannon diversity for a subfield is 100, it can be understood as saying that the subfield has the same fragmentation as an imaginary subfield with 100 tests that are all used equally often. Hill-Shannon diversities can be compared relatively, so a subfield with a value of 200 can be said to be twice as fragmented as a subfield with a value of 100.
June 13, 2025 at 10:50 AM
ruben.the100.ci
Previously, we had focused on the number of distinct measures/constructs (a measure ecologists call "richness") and used Shannon entropy to quantify fragmentation. But richness is easily biased by differential effort (any rare species you don't find decreases the number) and entropy is not intuitive
Number of distinct measures. Normalised Shannon entropy
June 13, 2025 at 10:50 AM
ruben.the100.ci
We could see this in the data, in 2016 the ~exponential growth stops. Screening out low effort contributions made sense to APA, but can we still infer how fragmentation changed when the database inclusion criteria changed so much?
June 13, 2025 at 10:50 AM
ruben.the100.ci
We worked with an existing database based on codings by APA staff. We know little about the internal process, but they did, for example, let us know, that around 2016 they were overwhelmed by new measures (interesting itself!), so they increased standards for what would make it into the database.
APA upped its standards in 2016, thereafter more measures had complete information about validity, factor structure etc. Number of items in measure over time in APA Psyctests
June 13, 2025 at 10:50 AM
ruben.the100.ci
Our paper "A fragmented field" has just been accepted at AMPPS. We find it's not just you, psychology is really getting more confusing (construct and measure fragmentation is rising).
We updated the preprint with the (substantial) revision, please check it out.
osf.io/preprints/ps...
Treemap showing measurement fragmentation across subfields in psychology. Hill-Shannon Diversity 𝐷=1626.05 How often measures in the APA PsycTESTS database are (re)used according to the APA PsycInfo database: rarely, the majority are never reused. Our fragmentation index (Hill-Shannon diversity) over time across subdisciplines shows fragmentation rising.
June 13, 2025 at 10:50 AM
ruben.the100.ci
Hadn't read this before and damn it goes hard.
Put succinctly, my argument is the following.Given the current state of affairs in psychology–in which 50% or moreof ourresults fail to replicate,thevastmajority of measures are ad hoc or of unknown validity,theories are so vague as to be mostly unfalsifiable, and thereis little to no overlap in approach between studies in supposedly related topical areas–neither meta-analysis nor any othercurrentlyknown consensus generating procedure is going to be sufficient to allow us to fashion gold from lead. Is my argumenta call for totally atheoretical, inductive work? No, because even descriptive work is theory-laden and must begin somewhere(Oreskes, 2019). But iftheories are explanations of available facts, then to strengthen our theories–and therefore our consensual understanding–it would be nice to first have some reliable facts to explain.
May 26, 2025 at 6:42 AM
ruben.the100.ci
Compare this to the much weaker statements that they actually found evidence to back up and got consensus around. I don't find this actionable except in the classic way: more and especially better research is needed.
my figure showing the actual statements they found consensus for from the final survey, including people who skipped claims
May 22, 2025 at 7:01 PM