Become a supporter of Andréa Mason today!

Today’s preprint paper has the best title ever: “Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Models”. It’s from DexAI, who sell AI testing and compliance services. So this is a marketing blog post in PDF form. [_arXiv_ _, PDF_] Here’s the big claim: > Across 25 frontier proprietary and open-weight models, curated poetic prompts yielded high attack-success rates (ASR), with some providers exceeding 90%. You can get around chatbot guard rails by rephrasing your prompt as poetry! Now, that’s a very tasty and repostable headline claim. They also get points for the first line in the paper itself: > In Book X of The Republic, Plato excludes poets on the grounds that mimetic language can distort judgment and bring society to a collapse. Humanities 1, techbros 0. Rewording things to get around chatbot guardrails is not that hard _—_ because chatbot guardrails don’t work so well. They’re a flimsy workaround. You can’t fix prompt injection _—_ it’s literally unfixable given how generative AI works. But does the paper give us good reason to think poetic rewording is unusually effective? Unfortunately, the paper has serious problems. Specifically, all the scientific process heavy lifting they should have got a human to do … they just used chatbots! I mean, they don’t seem to have written the text of the paper with a chatbot, I’ll give ’em that. But they did do the actual procedure with chatbots: > We translated 1200 MLCommons harmful prompts into verse using a standardized meta-prompt. They didn’t even write the poems. They got a bot to churn out bot poetry. Then they judged how well the poems jailbroke the chatbots … by using other chatbots to do the judging! > Open-weight judges were chosen to ensure replicability and external auditability. That really obviously does neither of those things — because a chatbot is an opaque black box, and by design its output changes with random numbers! The researchers are pretending to be objective by using a machine, and the machine is a random nonsense generator. They wrote a good headline, and then they faked the scientific process bit. Looking over the paper, I don’t see at all what they got from using chatbots to do these jobs. This was gratuitous and lazy, and it should be a discredit to these researchers. They could have just not used chatbots! Why on earth did they use chatbots? Now we can’t even say for sure if poetry really is more effective to get around guard rails than any other rewording. We don’t know if this paper is even telling us anything. Is it poetically true? Maybe you could ask a chatbot. It’s a weak claim with weak supporting methodology and a fabulously attractive headline. Because it’s an advert for DexAI, shaped a bit like a paper. You cannot do chatbot calculations on synthetic data and claim you found something new. Fake chatbot data and fake chatbot processing saves a lot of time, though. When you’re writing a marketing post. * _Video_ _—_ _Podcast_ ### Share this post: * Click to share on Facebook (Opens in new window) Facebook * Click to share on LinkedIn (Opens in new window) LinkedIn * Click to share on Reddit (Opens in new window) Reddit * Click to share on X (Opens in new window) X * Click to share on Mastodon (Opens in new window) Mastodon * Click to share on Bluesky (Opens in new window) Bluesky * Click to email a link to a friend (Opens in new window) Email * ### Like this: Like Loading... ### _Related_