P. Razavi
@p-razavi.bsky.social
Dad. Psych PhD. Research Scientist (Psychometrics, measurement).
Blog: Medium.com/@pooyar
Blog: Medium.com/@pooyar
Reposted by P. Razavi
Want to know what training data has been memorized by models like GPT-4?
We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models,
without requiring access to
🙅♀️ Model weights
🙅♀️ Training data
🙅♀️ Token probabilities 🧵 (1/5)
We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models,
without requiring access to
🙅♀️ Model weights
🙅♀️ Training data
🙅♀️ Token probabilities 🧵 (1/5)
Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models
High-quality training data has proven crucial for developing performant large language models (LLMs). However, commercial LLM providers disclose few, if any, details about the data used for training. ...
arxiv.org
March 21, 2025 at 7:08 PM
Want to know what training data has been memorized by models like GPT-4?
We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models,
without requiring access to
🙅♀️ Model weights
🙅♀️ Training data
🙅♀️ Token probabilities 🧵 (1/5)
We propose information-guided probes, a method to uncover memorization evidence in *completely black-box* models,
without requiring access to
🙅♀️ Model weights
🙅♀️ Training data
🙅♀️ Token probabilities 🧵 (1/5)
Reposted by P. Razavi
I’ve been referring people (esp social science/psych PhD students) to this blog post for years. The headline and opening paragraph are all you really need.
April 30, 2025 at 4:00 AM
I’ve been referring people (esp social science/psych PhD students) to this blog post for years. The headline and opening paragraph are all you really need.
Had a great time presenting our work on LLM-based item difficulty estimation at #NCME .
If you’re in Denver and would like to discuss measurement research or just catchup in the next couple of days, let me know 😊
If you’re in Denver and would like to discuss measurement research or just catchup in the next couple of days, let me know 😊
April 25, 2025 at 8:34 PM
Had a great time presenting our work on LLM-based item difficulty estimation at #NCME .
If you’re in Denver and would like to discuss measurement research or just catchup in the next couple of days, let me know 😊
If you’re in Denver and would like to discuss measurement research or just catchup in the next couple of days, let me know 😊
Fantastic, thoughtful work! 👏👏
April 18, 2025 at 4:32 PM
Fantastic, thoughtful work! 👏👏
I'm excited to share our latest work: "Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms." (🧵 1/9)
arxiv.org/abs/2504.08804
arxiv.org/abs/2504.08804
Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms
Estimating item difficulty through field-testing is often resource-intensive and time-consuming. As such, there is strong motivation to develop methods that can predict item difficulty at scale using ...
arxiv.org
April 17, 2025 at 2:33 AM
I'm excited to share our latest work: "Estimating Item Difficulty Using Large Language Models and Tree-Based Machine Learning Algorithms." (🧵 1/9)
arxiv.org/abs/2504.08804
arxiv.org/abs/2504.08804
Reposted by P. Razavi
Wooden Shoe Tulip Festival
📍Portland, Oregon 🇺🇲
📍Portland, Oregon 🇺🇲
April 10, 2025 at 1:30 PM
Wooden Shoe Tulip Festival
📍Portland, Oregon 🇺🇲
📍Portland, Oregon 🇺🇲
Reposted by P. Razavi
ネコ飛出し注意 (neko tobidashi chūi) Means “watch for cats darting out” and I love this sign.
April 7, 2025 at 3:17 AM
ネコ飛出し注意 (neko tobidashi chūi) Means “watch for cats darting out” and I love this sign.
Reposted by P. Razavi
A tricky thing about modern society is that no one has any idea when they don’t die.
Like, the number of lives saved by controlling air pollution in America is probably over 200,000 per year, but the number of people who think their life was saved by controlling air pollution is zero.
Like, the number of lives saved by controlling air pollution in America is probably over 200,000 per year, but the number of people who think their life was saved by controlling air pollution is zero.
April 7, 2025 at 4:13 AM
A tricky thing about modern society is that no one has any idea when they don’t die.
Like, the number of lives saved by controlling air pollution in America is probably over 200,000 per year, but the number of people who think their life was saved by controlling air pollution is zero.
Like, the number of lives saved by controlling air pollution in America is probably over 200,000 per year, but the number of people who think their life was saved by controlling air pollution is zero.
Reposted by P. Razavi
Did you know: our researchers have developed a suite of resources for A-Level students and teachers? "History & Philosophy of Science in 20 Objects" draws on an incredible array of items from our own collection ft. prompts, questions, videos and more! sway.cloud.microsoft/cEekCFBF5CGF... #histsci
HPS in 20 objects
This resource was produced by academics from the Centre for History and Philosophy of Science at the University of Leeds, where we have our Museum filled with artefacts that tell a stories about the H
sway.cloud.microsoft
April 4, 2025 at 9:25 AM
Did you know: our researchers have developed a suite of resources for A-Level students and teachers? "History & Philosophy of Science in 20 Objects" draws on an incredible array of items from our own collection ft. prompts, questions, videos and more! sway.cloud.microsoft/cEekCFBF5CGF... #histsci
Reposted by P. Razavi
rough (like uff in buff)
cough (like off in scoff)
drought (like ow in cow)
though (like o in no)
thought (like aw in saw)
through (like oo in woo)
Enough.
cough (like off in scoff)
drought (like ow in cow)
though (like o in no)
thought (like aw in saw)
through (like oo in woo)
Enough.
February 25, 2025 at 3:13 PM
rough (like uff in buff)
cough (like off in scoff)
drought (like ow in cow)
though (like o in no)
thought (like aw in saw)
through (like oo in woo)
Enough.
cough (like off in scoff)
drought (like ow in cow)
though (like o in no)
thought (like aw in saw)
through (like oo in woo)
Enough.
Reposted by P. Razavi
Hello to all my friends at SPSP seeing this message in a hallway or lobby as you hope you are staring at your phone with enough noticeable intensity to avoid having to interact with anyone
February 21, 2025 at 3:09 AM
Hello to all my friends at SPSP seeing this message in a hallway or lobby as you hope you are staring at your phone with enough noticeable intensity to avoid having to interact with anyone
Reposted by P. Razavi
1/3
Tutorial on exploring ecological momentary assessment data is online at AMPPS, with:
- Accessible ways to visualize data for better understanding
- Models to get some first insights
- Further reading boxes for more advanced topics
- Reproducible pipeline you can run over your own data
Tutorial on exploring ecological momentary assessment data is online at AMPPS, with:
- Accessible ways to visualize data for better understanding
- Models to get some first insights
- Further reading boxes for more advanced topics
- Reproducible pipeline you can run over your own data
February 13, 2025 at 12:04 PM
1/3
Tutorial on exploring ecological momentary assessment data is online at AMPPS, with:
- Accessible ways to visualize data for better understanding
- Models to get some first insights
- Further reading boxes for more advanced topics
- Reproducible pipeline you can run over your own data
Tutorial on exploring ecological momentary assessment data is online at AMPPS, with:
- Accessible ways to visualize data for better understanding
- Models to get some first insights
- Further reading boxes for more advanced topics
- Reproducible pipeline you can run over your own data
Reposted by P. Razavi
Some of us have been meeting up at SPSP for the last few years. This year marks our fifth gathering. Email one of us if you want to join! Location TBD.
@mdehghani.bsky.social @drsanaz.bsky.social @simine.com @dorsaamir.bsky.social
@mdehghani.bsky.social @drsanaz.bsky.social @simine.com @dorsaamir.bsky.social
February 13, 2025 at 3:33 PM
Some of us have been meeting up at SPSP for the last few years. This year marks our fifth gathering. Email one of us if you want to join! Location TBD.
@mdehghani.bsky.social @drsanaz.bsky.social @simine.com @dorsaamir.bsky.social
@mdehghani.bsky.social @drsanaz.bsky.social @simine.com @dorsaamir.bsky.social
Reposted by P. Razavi
My husband just inadvertently inspired one of the simplest, most relatable XY problem¹ demos I've seen.
He asked if I could buy unscented TP 🧻 next time I grocery shop.
Knowing he had been getting a cold, I probed: when does the scent become a problem?
[1/3]
¹ en.m.wikipedia.org/wiki/XY_prob...
He asked if I could buy unscented TP 🧻 next time I grocery shop.
Knowing he had been getting a cold, I probed: when does the scent become a problem?
[1/3]
¹ en.m.wikipedia.org/wiki/XY_prob...
XY problem - Wikipedia
en.m.wikipedia.org
February 5, 2025 at 4:01 PM
My husband just inadvertently inspired one of the simplest, most relatable XY problem¹ demos I've seen.
He asked if I could buy unscented TP 🧻 next time I grocery shop.
Knowing he had been getting a cold, I probed: when does the scent become a problem?
[1/3]
¹ en.m.wikipedia.org/wiki/XY_prob...
He asked if I could buy unscented TP 🧻 next time I grocery shop.
Knowing he had been getting a cold, I probed: when does the scent become a problem?
[1/3]
¹ en.m.wikipedia.org/wiki/XY_prob...
I’ll share a more detailed thread on this work later, but for now, I’m excited to share this preprint with the Blsky community! In this research, we used a range of methodologies including thematic analysis, closed- and open-vocabulary analyses (e.g., LIWC, topic modeling), and prototype approach...
Anger: Justified and Unjustified: http://osf.io/dxvbw_v1/
February 3, 2025 at 2:56 AM
I’ll share a more detailed thread on this work later, but for now, I’m excited to share this preprint with the Blsky community! In this research, we used a range of methodologies including thematic analysis, closed- and open-vocabulary analyses (e.g., LIWC, topic modeling), and prototype approach...
Reposted by P. Razavi
Lots of useful info in this thread if you are backing up public data (whether at OSF or elsewhere)
Hey if anyone is considering backing up any datasets for some reason today, the Open Science Framework has free storage up to 5G/private repository, and up to 50G/public repository.
(The OSF is run by a 501c3 (Center for Open Science), so if you do this you might also consider making a donation.)
(The OSF is run by a 501c3 (Center for Open Science), so if you do this you might also consider making a donation.)
January 31, 2025 at 8:06 PM
Lots of useful info in this thread if you are backing up public data (whether at OSF or elsewhere)
Reposted by P. Razavi
Reposted by P. Razavi
This thread shows that #measurement is essentially a series of compromises. We want comparability across time but also want metrics to reflect changes in priorities. We want psychometric properties but also input from stakeholders. We want translatability but also local validity. Can’t have it all
For anyone working in adolescent mental health: I *really* recommend reading this important paper about the most commonly used questionnaires (inc SDQ)
In short: most of them have poor psychometric properties, so do we even know what they are measuring?
www.thelancet.com/journals/lan...
(cont)
In short: most of them have poor psychometric properties, so do we even know what they are measuring?
www.thelancet.com/journals/lan...
(cont)
December 5, 2024 at 12:12 PM
This thread shows that #measurement is essentially a series of compromises. We want comparability across time but also want metrics to reflect changes in priorities. We want psychometric properties but also input from stakeholders. We want translatability but also local validity. Can’t have it all
Reposted by P. Razavi
Do you need educational and psychological item response data to do your research? The IRW has 600 item response datasets (more coming!), distributed in a standardized format and ready for analysis.
December 10, 2024 at 5:39 PM
Do you need educational and psychological item response data to do your research? The IRW has 600 item response datasets (more coming!), distributed in a standardized format and ready for analysis.
Reposted by P. Razavi
Here are some sayings we haven’t heard in a while:
-“Hold your horses!”
-“She really takes the egg.”
-“You’re the bee’s knees!”
-“Cash Money Records taking over for the 9-9 into 2000.”
-“Hold your horses!”
-“She really takes the egg.”
-“You’re the bee’s knees!”
-“Cash Money Records taking over for the 9-9 into 2000.”
January 6, 2025 at 7:02 PM
Here are some sayings we haven’t heard in a while:
-“Hold your horses!”
-“She really takes the egg.”
-“You’re the bee’s knees!”
-“Cash Money Records taking over for the 9-9 into 2000.”
-“Hold your horses!”
-“She really takes the egg.”
-“You’re the bee’s knees!”
-“Cash Money Records taking over for the 9-9 into 2000.”
Reposted by P. Razavi
Posting again since I’m trying to get my syllabus together: I’m teaching objective assessment of personality next semester. Any strong recs for things to cover and main take away points?
December 31, 2024 at 2:32 PM
Posting again since I’m trying to get my syllabus together: I’m teaching objective assessment of personality next semester. Any strong recs for things to cover and main take away points?
Reposted by P. Razavi
For all intensive purposes, what idioms or colloquialisms do people seem to constantly get wrong?
December 30, 2024 at 2:12 PM
For all intensive purposes, what idioms or colloquialisms do people seem to constantly get wrong?
