Threat model
Where an agent can be attacked or go wrong: untrusted text, sensitive access and outward actions, plus the public risk catalogues to check yourself against.
Start with the question “what can this agent reach?”
A threat model for an agent is short. List what the agent can read, what it can change, and which text it reads that someone else controls. Each of those is a place where things can go wrong, whether through an attacker, a mistake, or both.
Diagram: my own.
The key property is that a language model follows instructions it finds in content, not only instructions from its operator1. So every source in the diagram above is an entry point, not just the user’s chat box. See Prompt injection for how that plays out.
The lethal trifecta
Simon Willison named a combination of three capabilities that makes data theft easy: access to private data, exposure to untrusted content, and a way to communicate externally1. If an agent has all three, an attacker who can get text in front of it may be able to make it read private data and send it out1.
Diagram: my own, after the framing in 1.
Willison notes that when vendors found this kind of problem in their products they usually fixed it by closing the exfiltration route1. That works for one product. It does not help when you combine tools yourself, because then the combination is your design decision1.
The rule of two
Meta’s security team turned this into a design rule. Their Agents Rule of Two says that, until prompt injection can be reliably detected and refused, an agent should satisfy no more than two of three properties in one session: it processes untrusted input, it has access to sensitive systems or private data, and it can change state or communicate externally2. Willison points out that this adds “changing state” as a property the trifecta did not cover3.
Diagram: my own, after 2. The bottom box paraphrases Meta’s advice to add supervision or another reliable check when all three are needed.
This is a good first review question for any new agent. It is not a proof of safety, and Meta frames it as a minimum to design around2.
Public catalogues to check against
You do not need to invent a taxonomy.
- OWASP Top 10 for LLM Applications (2025). Prompt injection is first, followed by sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption4. The 2025 edition expanded excessive agency in response to agentic designs5.
- OWASP Top 10 for Agentic Applications (2026). Published in December 2025 by the OWASP GenAI Security Project, with input from more than 100 experts, for systems that plan, use tools and act6. Its entries are numbered ASI01 to ASI106.
- NIST AI 600-1. A generative AI profile that sits alongside NIST’s AI Risk Management Framework and names twelve risks specific to or made worse by generative AI7. It is voluntary guidance.
A worksheet (my own approach)
- List every tool and what it can read or change.
- List every source of text the model reads, and mark which an outsider can influence.
- Mark which tools can send data out, directly or through a URL.
- Check the rule of two. If all three hold, redesign or add an approval step (Human approval).
- For each remaining risk, pick a control: permissions, sandboxing, guardrails, or monitoring (Tracing and monitoring).
- Test the result with red teaming.
Revisit the worksheet whenever you add a tool or a data source. Most agent incidents come from a change to what the agent can reach.
Sources
- [1]The lethal trifecta for AI agents: private data, untrusted content, and external communication — Simon Willison · 2025PARAPHRASED
- [2]Agents Rule of Two: A Practical Approach to AI Agent Security — Meta AI · 2025PARAPHRASED
- [3]New prompt injection papers: Agents Rule of Two and The Attacker Moves Second — Simon Willison · 2025PARAPHRASED
- [4]OWASP Top 10 risks for LLMs (2025 update) — Zbigniew Banach, Invicti · 2025LIST OF ENTRIES, AS SUMMARISED BY A THIRD PARTY
- [5]OWASP Top 10 for LLM Applications 2025 — AI Governance Library (summary of the OWASP guidance)CHANGES IN THE 2025 EDITION, AS SUMMARISED BY A THIRD PARTY
- [6]OWASP Top 10 for Agentic Applications for 2026 — OWASP GenAI Security Project · 2025PARAPHRASED
- [7]NIST AI 600-1 Generative AI Profile guide — Anaconda (summary of NIST AI 600-1)OVERVIEW OF THE PROFILE, AS SUMMARISED BY A THIRD PARTY