Prompt injection
Why models obey instructions hidden in content, what attackers do with that, and which defences change the outcome rather than just lowering the odds.
What it is
Prompt injection is when text the model reads changes what it does, against the wishes of the person or system that deployed it. It matters most for agents, because an agent can act on what it was told.
Diagram: my own.
- Direct: the person typing is the attacker, for example trying to override a system prompt.
- Indirect: the attacker plants instructions in content the agent will fetch. The 2023 paper by Greshake and colleagues introduced the term and showed attacks on real systems including Bing Chat and code-completion tools1.
Their central argument is that retrieval blurs the line between data and instructions1. Their taxonomy covers data theft, worming, contamination of the information ecosystem and other impacts1.
A concrete example
Willison’s illustration is mundane. You ask an assistant to summarise a web page, and the page contains text telling the assistant to retrieve your private data and email it to an attacker. There is a good chance the model complies2. He lists many shipped products where variants of this were found, almost all of which vendors then fixed by closing the exfiltration route2.
Why it is hard
The model receives trusted instructions and untrusted content in the same stream of tokens. Training and prompting can teach it to be sceptical, but that is a statistical defence. The authors of the design-patterns paper argue that improvements in detection and agent capability are likely to stay heuristic and so brittle, and that general-purpose agents are therefore unlikely to give reliable safety guarantees with today’s models3.
Defences, from strongest to weakest
1. Limit what a hijacked agent can do. Apply the rule of two, least-privilege permissions and a sandbox. These work even when the model is fooled.
2. Constrain the design. The design-patterns paper proposes agent designs that deliberately restrict the agent so it cannot solve arbitrary tasks, trading generality for resistance to injection, and works through ten case studies3. CaMeL goes further: it wraps the model in a system layer that extracts control flow and data flow from the trusted request, so untrusted data cannot change the program flow, and it tracks capabilities on values to stop leaks4. It uses a privileged model that plans and a quarantined model that handles untrusted data4. On the AgentDojo benchmark, it solved 77 percent of tasks with provable security, against 84 percent for an undefended system4.
Diagram: my own simplification in the spirit of 3 and 4; it is not the exact architecture of either.
3. Train and test the model. Anthropic describes exposing Claude to injections in simulated web content during training and rewarding refusal5. They report that, against an internal adaptive attacker, the browser extension with their newest model reached roughly a 1 percent attack success rate, and say that this still represents meaningful risk and that no browser agent is immune5. Their earlier pilot tested 123 cases across 29 attack scenarios and found concerning results before mitigations6. Treat vendor numbers as indicative: they come from the vendor’s own tests.
4. Detect and filter. Classifiers and screening prompts catch some attacks (Input and output guardrails). They help, but a filter that misses a few percent of attacks is not a boundary.
What to do in practice
- Treat all retrieved, fetched and tool-returned text as untrusted, and say so in your prompts, knowing that this alone is weak.
- Remove outbound channels you do not need, including links and images that load from attacker-chosen URLs (my own advice).
- Keep sensitive tools out of sessions that read untrusted content, or require human approval before they run.
- Add injection cases to your eval set and run them with every change (Red teaming).
- Log tool calls so you can see what an injected agent actually did (Tracing and monitoring).
Sources
- [1]Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Greshake, Abdelnabi, Mishra, Endres, Holz, Fritz · 2023PARAPHRASED
- [2]The lethal trifecta for AI agents: private data, untrusted content, and external communication — Simon Willison · 2025PARAPHRASED
- [3]Design Patterns for Securing LLM Agents against Prompt Injections — Beurer-Kellner et al. · 2025PARAPHRASED
- [4]Defeating Prompt Injections by Design — Debenedetti et al., Google and ETH Zurich · 2025PARAPHRASED
- [5]Mitigating the risk of prompt injections in browser use — Anthropic · 2025PARAPHRASED
- [6]Piloting Claude in Chrome — Anthropic · 2025PARAPHRASED