Threat Modelling an LLM System
The trust boundaries specific to model-based applications, an inventory of assets and adversaries worth enumerating, and how to turn that into controls rather than a document.
Standard threat modelling assumes components with defined interfaces and predictable behaviour. An LLM application has a component whose behaviour is probabilistic, whose instructions arrive in the same channel as its data, and whose failures are not exceptions but plausible-looking output. The method still applies; the boundaries and the assets are different.
Trust boundaries specific to these systems
User input to the prompt. Direct injection, and the point where the application's own instructions meet the user's.
Retrieved content to the prompt. The indirect injection boundary, and usually the highest-severity one, because the content's author is not the user and may be anyone.
Tool results to the prompt. Frequently forgotten. A tool returning attacker-influenced data, an API response, a database row containing user-supplied text, a file's contents, is an injection vector identical to retrieval.
Model output to actions. Where a generated tool call becomes a real effect, and where the reversibility test applies.
Model output to the user. Where hallucination, harmful content, and rendering-based exfiltration land.
Application to model provider. Where prompts, which contain user data, leave the trust domain, with its own contractual, retention and jurisdictional questions.
Assets and adversaries
Assets worth listing: the system prompt and any business logic in it, user data in prompts and conversation history, credentials and tokens accessible to tools, the model itself if self-hosted, the compute budget, and the organisation's reputation for what the system says.
Adversaries worth distinguishing: the user attacking the system for their own benefit; a third party planting content the system will read; an attacker targeting other users through shared state such as a poisoned index; and an insider with access to prompts or logs. Each reaches different boundaries, and conflating them produces controls aimed at the wrong one.
From model to controls
The output of a threat model should be controls with owners, not a diagram. The mapping that tends to hold: injection risks map to permission scoping and egress restriction rather than to prompt hardening; data leakage maps to logging policy, retention and redaction; cost and abuse map to quotas and rate limits; output harm maps to filtering, human review and disclosure.
Reviewing the model when the capability set changes matters more than reviewing it on a schedule, because adding one tool can change the severity of every existing injection path.
When it breaks
The model is not a security control. Prompt instructions telling the model to refuse are a usability feature with security-adjacent effects. Any control that must hold has to be enforced outside the model, in code that cannot be argued with.
Aggregation changes severity. A tool that returns one record is low risk; one that can be called a thousand times in a loop is a bulk export. Rate and volume limits are part of the permission model, not an operational afterthought.
Shared state creates cross-user attacks. A vector index, a cache or a memory store written by one user and read by another turns a single-user injection into a persistent multi-user compromise. Anything shared and writable is a boundary.
The provider is in the trust domain. Prompts contain whatever users typed. Retention, training use, and jurisdiction are properties of the contract rather than of the code, and they belong in the threat model rather than being treated as procurement's problem.
12 flashcards for this concept
Click a card to reveal the answer.