phelimb
phe-lim.bsky.social
phelimb
@phe-lim.bsky.social
CEO & Founder @ Prolific.com
All fair. Expected to see more diversity in modalities also. Qual studies (which can now be done at scale) are likely to be more robust than survey only.
November 19, 2025 at 4:08 PM
Studies aren't distributed on a first-come first-served basis, but it's a useful theory. I will share with the team.
November 19, 2025 at 2:26 PM
Right. It is generalisable (JS plugin), though we don't have a native integration with otree yet.
November 19, 2025 at 2:10 PM
Will reach out to the authors to see if we can understand more details & see if we can add Authenticity Check as a mitigation option.
November 19, 2025 at 2:08 PM
45% of participants copying OR pasting ~= 45% LLM use.

Only single-digit responses seem to fail their honeypot and other mitigations, which is closer to our internal prevalence measures.
November 19, 2025 at 2:06 PM
There are many reasons to copy/paste while still being a conscientious human.

"Even to an untrained eye,some of these responses were obviously generated by LLMs", but the % doesn't seem reported?
November 19, 2025 at 2:04 PM
If I'm reading the paper correctly, their detection of prevalence was "we only tracked copying and pasting on a page containing an openended question" - this is fairly crude measure of llm detection and is upper bound rather than an accurate prevalence measure
November 19, 2025 at 2:03 PM
LLM use by real humans is a slightly different threat to the scaled agent threat discussed in the paper though, and I think requires a bit more nuance in its response.
November 19, 2025 at 11:31 AM
I hadn't, thanks for sharing. Agree with many of the mitigation strategies, though given data was collected on Prolific we would have reccommeded our built in tool.

researcher-help.prolific.com/en/articles/...
How to add authenticity checks to your Qualtrics study | Prolific Research
researcher-help.prolific.com
November 19, 2025 at 11:30 AM
prolific.com/resources/pr...

If you want to work on these problems, or collaborate on research in this area, get in touch. Much more to come in this space!
Prolific sets standards for authentic human data collection | Prolific
Discover how Prolific's data quality system, Protocol, sets industry standards for authentic human data collection
prolific.com
November 19, 2025 at 11:24 AM
We also do spot checks to protect against account reselling: participant-help.prolific.com/en/articles/...
Why have I been asked to recheck my identity? | Prolific Participants
participant-help.prolific.com
November 19, 2025 at 11:17 AM
Not sure I agree – these are tractable challenges and we are working on them.

bsky.app/profile/phe-...
This is a super important paper, but honestly, I think the opposite. Integrity and authenticity of online-based panels have been in question prior to this risk agents. IMO this forces all providers to up their verification and transparency of their participants.e.g.
www.prolific.com/resources/pr...
Prolific sets standards for authentic human data collection | Prolific
Discover how Prolific's data quality system, Protocol, sets industry standards for authentic human data collection
www.prolific.com
November 19, 2025 at 11:12 AM
Tara, let me know if I can share any more info that would put your mind at ease. We have implemented numerous mitigations to ensure the collection of authentic data (and continue to assess and invest further).
November 19, 2025 at 10:18 AM
www.prolific.com/resources/in...

There's a lot of work to do here, though, and it's likely to continue being a bit of an arms race. Our bet is on many layers of assessment here, but we are keen to collaborate with researchers & the broader community to address this (very real) threat.
New authenticity checks detect AI misuse in research | Prolific
Introducing authenticity checks: Ensure genuine human responses in the age of AI
www.prolific.com
November 18, 2025 at 9:26 PM
Agreed that this is the harder challenge, though we do have tooling for this also (in addition to throttling). Works with Qualtrics.
New authenticity checks detect AI misuse in research | Prolific
Introducing authenticity checks: Ensure genuine human responses in the age of AI
www.prolific.com
November 18, 2025 at 9:25 PM
& we're continuing to invest here. I'd be happy to work with you or Sean on replication if of interest.

Some notes on our continued identity verification and agent detection here:
participant-help.prolific.com/en/articles/...
www.prolific.com/resources/in...
November 18, 2025 at 9:19 PM
I disagree. AFAIK, this was not run on Prolific, and we already have the majority of Sean's recommendations in place.
Ongoing Panelist Validation ✅
Throttling Mechanisms ✅
Panelist Professionalism ✅
Panelist Quality Checks ✅
Location Checks ✅
Identity Validation ✅
Secure Software 🟠 (partial)
November 18, 2025 at 9:16 PM
This is a super important paper, but honestly, I think the opposite. Integrity and authenticity of online-based panels have been in question prior to this risk agents. IMO this forces all providers to up their verification and transparency of their participants.e.g.
www.prolific.com/resources/pr...
Prolific sets standards for authentic human data collection | Prolific
Discover how Prolific's data quality system, Protocol, sets industry standards for authentic human data collection
www.prolific.com
November 18, 2025 at 9:03 PM