Agents that read untrusted content and call tools are a new class of attack surface. Treat every tool call as if the model were a partially-trusted user.
Top risks (OWASP LLM Top 10)
- LLM01 — Prompt injection (OWASP) — the model follows instructions hidden in retrieved documents, emails, or web pages.
- LLM02 — Insecure output handling — using model output directly in shell, SQL, or HTML.
- LLM06 — Excessive agency — the agent has tools with broader permissions than the task requires.
Defenses
- Least privilege — scope tool credentials to the user and the action. Never give one agent a global admin key.
- Human-in-the-loop for irreversible actions (payments, deletes, sends).
- Input isolation — clearly delimit untrusted content (
<untrusted>...</untrusted>) and instruct the model to never follow instructions inside. - Output validation — parse and validate tool arguments before executing.
- Rate + spend limits per user and per session.
Deeper reading
- Simon Willison — Prompt injection
- NIST — AI Risk Management Framework
- Anthropic — Agent capabilities and safety