Tool Authorisation Boundary

A model given untrusted text cannot reliably tell instructions from data. Text arriving as data — a user message, a retrieved document, a web page, an email — may contain instructions, and the model may follow them. When the model can call tools, those instructions become actions.

Prompt-level defences are not reliable. Instructions telling the model to ignore instructions in the input are themselves text in the same channel, and every published mitigation of that form has been bypassed.

The control belongs in the authorisation layer around the tools.

Why it matters

It is the difference between an injection being an inconvenience and being a breach. It also produces a system whose security properties can be reasoned about without reasoning about the model's behaviour, which is the only tractable position.

A system whose security depends on the model behaving correctly has no security boundary, and model improvements do not change that.

Implementation patterns

• Every tool call authorised against the initiating user's permissions, evaluated outside the model. If the user cannot perform the action, the call fails regardless of what the model attempted.
• Minimum tool scope, with destructive or irreversible operations not exposed to a model acting on untrusted input at all.
• Explicit confirmation for consequential actions, described to the user in the application's own words rather than the model's — since the model's description is itself potentially influenced.
• Untrusted content marked as untrusted throughout the pipeline, with content from third parties never given the trust of the user's own instruction.
• Model output treated as untrusted wherever it is rendered, executed or used in a query. Conventional injection defences apply and are frequently forgotten in this context.
• Corpus ingestion treated as a security boundary, because a poisoned document in the retrieval index is an injection reaching every user whose query retrieves it — so user submissions, crawled sources and partner feeds need validation before indexing, with provenance visible in the retrieved context.
• Injection detection as a mitigation, not a control. It is probabilistic and must never be the thing standing between an attacker and an irreversible action.

Industry example

Platforms such as Hasura that generate and execute queries from declarative input face the general form of this problem: an intermediary translating intent into a privileged operation must authorise the operation independently of the intent's source. The AI case is the same shape with a less predictable translator.

Failure scenarios

• Authorisation implemented in the system prompt, which is text an attacker can address.
• Tools running with the application's permissions rather than the user's.
• Irreversible operations exposed to a model handling untrusted content.
• Retrieval corpora ingesting unvalidated third-party content.
• Model output rendered without escaping, reintroducing conventional injection.
• Detection treated as prevention.

Trade-offs

Per-user authorisation on every tool call adds latency and requires a permission model fine-grained enough to express it — which many applications do not have and which is real work to build.

The alternative is a capability that is either unusable (no tools) or unsafe (tools with ambient privilege), and the middle position of "tools with careful prompts" is the one that appears safe and is not. Building the permission model is the cost of having agents that act on real systems.

Interview question

"Your assistant can read a customer's emails and can send emails. An attacker emails the customer with instructions embedded in the message. Walk me through what stops it, and tell me which of your defences you would rely on if the others failed."