Capability Confinement

An agent with tool access reads content it did not author: documents, web pages, tickets, emails, API responses. That content can contain instructions, and the model has no reliable mechanism to distinguish instructions from data — both arrive as tokens in one context, and the model's training is precisely to follow instructions found in text.

Prompt-layer mitigations reduce the success rate of injection and do not eliminate it. They are probabilistic defences against an adversary who can iterate, which makes them useful and disqualifies them as controls.

Capability confinement accepts this. Assume the model will sometimes be misled, and ensure that being misled cannot cause serious harm — by constraining what actions are available and what authority they carry.

Why it matters

The model is not a security boundary and cannot be made into one. Every design whose safety depends on the model refusing a cleverly-worded instruction is, at present, unsafe — and the industry has repeatedly demonstrated this with each new mitigation.

The reframing produces the right questions immediately: what can this agent do, on whose authority, with what confirmation, and what is the worst outcome if every instruction it follows was written by an attacker? Those are answerable by ordinary access control, which is a mature and reliable discipline, rather than by model behaviour, which is not.

Implementation patterns

• Minimum tool set per task, and narrowing it the moment untrusted content enters the context. An agent summarising a document does not need to send email.
• Trust domain separation: one agent reads untrusted content with no tool access and emits structured output; another acts on that structured output. The boundary between them carries data rather than instructions, which is the strongest practical pattern available.
• Human confirmation for consequential and irreversible actions — payments, external messages, deletions, permission changes — showing what will actually happen rather than a model-generated summary, since a summary can itself be manipulated.
• Schema and range validation on every tool call, so parameters must conform to expected shapes.
• Egress restriction. An agent that cannot make arbitrary outbound requests cannot exfiltrate regardless of what it is persuaded to do — one of the most effective and least applied controls.
• Actions executed with the user's authority, not the agent's, so the agent can never exceed what the person it acts for could do themselves.
• Provenance marking of untrusted content in the context, so policy can restrict what is permitted after it has been read.
• Per-session spend and rate caps, bounding the damage of a successful injection.
• Complete tool-call logging with inputs, so injections are detectable afterwards.

Industry example

The pattern has become standard guidance as agentic products have shipped: published guidance from model providers and security researchers converges on least privilege, human confirmation for consequential actions, and separating the component that reads untrusted content from the component that acts — because every proposed prompt-level defence has subsequently been bypassed.

The recurring real-world demonstrations are indirect: an agent reading a web page, an email, a code repository or a support ticket containing instructions, and taking an action the user never requested. The successful mitigations in each case were architectural — the agent could not reach the resource, or a human had to confirm — never prompt-level.

Failure scenarios

• A broad tool set available at all times, regardless of what the agent is currently doing.
• Untrusted content and high-privilege capabilities in the same context.
• Confirmation on a model-generated summary rather than on the concrete action.
• The agent holding standing credentials broader than the user it acts for.
• Unrestricted egress, permitting exfiltration through any successful injection.
• Relying on the model to refuse, and treating a refusal-tuned model as a control.
• No logging of tool calls, so a successful injection is undetectable.
• Confirmation fatigue — so many prompts that users approve reflexively, which converts the control into a formality.

Trade-offs

Confinement directly limits usefulness. The value of an agent is largely in what it can do autonomously, and every confirmation step and removed capability reduces that. An agent that must ask before every action is a slower way to do the task manually, and users notice.

There is also a genuine tension in the confirmation design: too many prompts produce reflexive approval, too few leave consequential actions unguarded, and the calibration is per-action rather than global — irreversible and externally-visible actions warrant a prompt; reversible internal ones do not.

The trade is autonomy and convenience in exchange for a bounded worst case. For a read-only agent operating on trusted content the confinement can be light. For an agent with payment, communication or deletion capability processing anything a third party can influence, it is the only thing standing between a product and a serious incident — and the correct calibration is set by consequence, not by capability.

Interview question

"Our agent reads customer support tickets and can issue refunds. A ticket contains text instructing it to refund a large amount to a specified account. Tell me every control that should have stopped that, and which single one you would implement first if you only had a week."