Giacomo Bignardi
@bignardi.bsky.social
Research Associate at the Social, Genetic & Developmental Psychiatry Centre, Kings College London. Interests: developmental psychology, data science, coffee. www.bignardi.co.uk
Dont forget the true score isnt the actual "true" score, but the expected score.
November 7, 2025 at 7:14 PM
Dont forget the true score isnt the actual "true" score, but the expected score.
Link doesn't work for me sadly! :( Says bad gateway error?
October 29, 2025 at 12:07 PM
Link doesn't work for me sadly! :( Says bad gateway error?
Interesting, thanks! I need to read up on this again as I don't remember all the different ICCs. Cool that it seems to match up with this formula!
October 20, 2025 at 7:36 PM
Interesting, thanks! I need to read up on this again as I don't remember all the different ICCs. Cool that it seems to match up with this formula!
... which would converge to 1 as the number of raters increases? I assume with inter-rater reliability, researchers are interested in something like how well a single rater's ratings of the items match the "true" rating, which is quite different, and necessitates a different approach?
October 20, 2025 at 5:46 PM
... which would converge to 1 as the number of raters increases? I assume with inter-rater reliability, researchers are interested in something like how well a single rater's ratings of the items match the "true" rating, which is quite different, and necessitates a different approach?
Hmmm - I haven't really thought about inter-rater reliability since 2016 - so I'm not sure how RMU would map onto existing metrics yet. Thinking aloud: applying RMU to the item intercepts gives the reliability of the model's item estimates when combining all information across all raters...
October 20, 2025 at 5:46 PM
Hmmm - I haven't really thought about inter-rater reliability since 2016 - so I'm not sure how RMU would map onto existing metrics yet. Thinking aloud: applying RMU to the item intercepts gives the reliability of the model's item estimates when combining all information across all raters...
Could you share the models/code your using? Do you have estimates for subjects that you plug into the function?
October 17, 2025 at 1:11 PM
Could you share the models/code your using? Do you have estimates for subjects that you plug into the function?
Neat! I have brms code for replicating (and going beyond) alpha too incase its useful: www.bignardi.co.uk/8_bayes_reli...
Estimating Mean Score Reliability with RMU
www.bignardi.co.uk
October 10, 2025 at 9:26 PM
Neat! I have brms code for replicating (and going beyond) alpha too incase its useful: www.bignardi.co.uk/8_bayes_reli...
Hmm, interesting, I've had this on my reading list but have not read it yet, the intro is so clear and great! I disagree that reliability implies equal precision across scores, but it seems like an interesting way of going beyond reliability...
October 10, 2025 at 9:12 PM
Hmm, interesting, I've had this on my reading list but have not read it yet, the intro is so clear and great! I disagree that reliability implies equal precision across scores, but it seems like an interesting way of going beyond reliability...
Good point, would be interesting to try (and also force myself to learn generalizability theory in more detail)!
October 10, 2025 at 8:30 PM
Good point, would be interesting to try (and also force myself to learn generalizability theory in more detail)!
Helping a PhD student with SDT models was actually the original inspiration for this side project! I have example code applying the method to a SDT model below (actually borrowing code from @matti.vuorre.com's helpful blog on this) ->
Tutorial: Calculating RMU reliability for a go/no-go task
www.bignardi.co.uk
October 10, 2025 at 8:04 PM
Helping a PhD student with SDT models was actually the original inspiration for this side project! I have example code applying the method to a SDT model below (actually borrowing code from @matti.vuorre.com's helpful blog on this) ->
Ah i misunderstood, no introduction rewrite needed yet then 😅. Good to know it gives similar results to the other approaches!
October 10, 2025 at 3:51 PM
Ah i misunderstood, no introduction rewrite needed yet then 😅. Good to know it gives similar results to the other approaches!
Exciting! Just to check - can you get confidence intervals from PSI & EAP too?
October 10, 2025 at 3:33 PM
Exciting! Just to check - can you get confidence intervals from PSI & EAP too?
Thanks Edwin! I only learned from the best 😜
October 1, 2025 at 11:52 AM
Thanks Edwin! I only learned from the best 😜
Great! Sorry just saw this before my replies :)
October 1, 2025 at 11:40 AM
Great! Sorry just saw this before my replies :)
I think both methods should converge to the same answer as the number of draws -> ∞. I think your approach is 1-V_w/V whereas empirical reliability is defined as v_a/(v_a+v_w) - but it should be the same as draws -> ∞. Pic from onlinelibrary.wiley.com/doi/10.1002/...
October 1, 2025 at 11:38 AM
I think both methods should converge to the same answer as the number of draws -> ∞. I think your approach is 1-V_w/V whereas empirical reliability is defined as v_a/(v_a+v_w) - but it should be the same as draws -> ∞. Pic from onlinelibrary.wiley.com/doi/10.1002/...
Hi Ruben - i had the same thought - and it wasn't easy finding the reference in an old IRT manual! In your post, do you divide the variance of subjects' posterior means by the total variance in MCMC draws across subjects? The best description of ER is in here by Phil Chalmers tinyurl.com/4f3vv5e8
Difference between empirical and marginal reliability of an IRT model
I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the
tinyurl.com
October 1, 2025 at 11:21 AM
Hi Ruben - i had the same thought - and it wasn't easy finding the reference in an old IRT manual! In your post, do you divide the variance of subjects' posterior means by the total variance in MCMC draws across subjects? The best description of ER is in here by Phil Chalmers tinyurl.com/4f3vv5e8
Any (ideally helpful) comments/feedback/critique welcome! Email at: bignardig@outlook.com
October 1, 2025 at 8:17 AM
Any (ideally helpful) comments/feedback/critique welcome! Email at: bignardig@outlook.com
Alongside the preprint osf.io/h54k8 I've uploaded an example of how to calculate RMU with go/no-go task data, comparing RMU with split-half and test-retest reliability estimates www.bignardi.co.uk/8_bayes_reli... - more software support to come if there's demand.
Tutorial: Calculating RMU reliability for a go/no-go task
www.bignardi.co.uk
October 1, 2025 at 8:17 AM
Alongside the preprint osf.io/h54k8 I've uploaded an example of how to calculate RMU with go/no-go task data, comparing RMU with split-half and test-retest reliability estimates www.bignardi.co.uk/8_bayes_reli... - more software support to come if there's demand.
In the paper, we also demonstrate that our method yields a similar point estimate to using the ratio of subjects' posterior mean variance (PMV) divided by PMV + average posterior variance. However, our approach also provides credible intervals, which are essential when sample sizes are small.
October 1, 2025 at 8:17 AM
In the paper, we also demonstrate that our method yields a similar point estimate to using the ratio of subjects' posterior mean variance (PMV) divided by PMV + average posterior variance. However, our approach also provides credible intervals, which are essential when sample sizes are small.