@cameron.pfiffer.org planning to work on it soon!

hi @alt.psingletary.com! you tagged the right person—I was working on this for a class project this semester

got it to a mvp stage about a week ago and hit pause to work on some other projects, but will keep working on it and would definitely would love to hear your feedback if you have any :)

and please remember to thank your local site reliability engineer!!!!

bsky.app/profile/haro...

There's a quickly-developing line of work on how insecure these agent systems can be, particularly when they have access to write and execute code.

The attacks on them are simple + devastating, up to and including reverse shells, data exfiltration, and more!

arxiv.org/abs/2503.12188

Anyhow, there’s a lot more in the paper. Please read it if you’re interested and let us know if you have any thoughts, questions, concerns, etc!

arxiv.org/abs/2503.12188

12/12

Modern Web browsers isolate untrusted content using the same-origin policy. AI agents today do not distinguish safe from unsafe content, nor data from (potentially malicious) instructions.

developer.mozilla.org/en-US/docs/W...

en.wikipedia.org/wiki/Same-or...

11/12

The narrative around AI safety shouldn’t be “Terminator” or “AI Chernobyl.” The right analogy is Netscape Navigator 1.0—the era when Web browsers first became a thing, and it was unclear how to protect users from potentially harmful Web content.

10/12

Much of the AI safety world is obsessing about “AGI.” They research containment, alignment, and jailbreaking, and view users as potential adversaries.

But users aren’t the enemy. They are victims whose data and devices are put at risk by companies pushing insecure systems.

9/12

At the root, these are “confused deputy” vulnerabilities: agents blindly trust other agents, enabling adversaries to launder their instructions by making them appear as trusted outputs of trusted agents.

en.wikipedia.org/wiki/Confuse...

8/12

In our experiments, we saw cases where a MAS …
… executes code that they recognize as harmful
… automatically pivots to harmful tasks that are simply in the same directory as benign tasks
… is vulnerable to screenshots and even audio files where we read out the attack (see example below⬇️⬇️⬇️)

7/12

These attacks are effective …
… across multiple agent frameworks (we tested AutoGen, MetaGPT, Crew AI), orchestrators, and LLMs
… even when direct and indirect prompt injection attacks don’t work
… even when individual agents are “aligned” and refuse to take harmful actions

6/12

This attack is simple and deadly (and multi-modal, too!): an attacker puts up a static webpage and lures a MAS to it. Without any user involvement, the page gets the MAS to run arbitrary malicious code on the user’s device or container, giving the attacker full control.

5/12

MASes rely on control flow processes: agents exchange metadata (status reports, error messages, etc.) to jointly plan and fulfill tasks on users’ behalf. Our paper demonstrates how adversarial content can hijack these processes to stage devastating attacks.

4/12

But not all internet content is trustworthy and safe. Adversaries can put up webpages and social media posts, send emails with attachments, etc. – all of which will be processed by a MAS. These systems will inevitably encounter malicious, adversarial content.

arxiv.org/abs/2503.12188

3/12

LLM agents are all the rage. Multi-agent systems (MAS) are promising a future where people interact with the internet via commands to semi-autonomous agents. Frameworks like AutoGen, Crew AI, and MetaGPT already enable developers to do this.

arxiv.org/abs/2503.12188

2/12