LLM Application Security
Injection across trust boundaries, tool and sandbox escape, secret exposure and threat modelling for agents.
12concepts
108flashcards
88minutes of reading
- 01 Agent Identity and Workload Credentials Why an agent is a third party rather than a proxy for the user, what goes wrong when it authenticates with a shared long-lived key, and how workload identity replaces that key with an attested, short-lived credential that names the running process.
- 02 Human-in-the-Loop Authorisation Design Where to put the human in an agent's action path so that approval remains a real decision, why per-action prompts decay into rubber-stamping, and how budgets, reversibility tiers and batched review preserve the signal.
- 03 Secrets and Credentials in LLM Context Why anything placed in a model's context should be treated as disclosed to whoever can influence its output, how credential brokering keeps secrets out of the prompt entirely, and where secrets arrive uninvited through users, logs and indexed tools.
- 04 Agent Permissions and Blast Radius How to design tool access for a component you must assume is compromised, why capability scoping beats identity-based permissions here, and the reversibility test that decides what needs confirmation.
- 05 Audience Binding and Token Exchange Why a bearer token that does not name its destination becomes a skeleton key the moment an agent passes it along, and the OAuth mechanisms — resource indicators, token exchange, protected resource metadata — that MCP now requires to stop it.
- 06 Delegated Authority and the Confused Deputy The 1988 result that a privileged intermediary acting on ambiguous instructions will misuse its own authority, why every tool-using agent is that intermediary, and the capability discipline that fixes it where filtering cannot.
- 07 Output Handling and Downstream Injection Why model output is untrusted input to everything it touches, the injection classes that follow from rendering or executing it, and the encoding discipline that prevents them.
- 08 Prompt Injection as a Trust Boundary Failure Why injection is architectural rather than a prompting problem, the distinction between direct and indirect injection that decides severity, and why no known technique closes it.
- 09 RAG Poisoning and Retrieval Trust How a handful of crafted passages in a corpus of millions can control what a RAG system says, why detection-based defences have fallen short, and why the durable controls are provenance on the write path and aggregation that bounds any single passage's influence.
- 10 Red-Teaming and Security Evaluation for LLM Apps What automated attack generation covers, why a pass rate is not a security property against an adversary who retries, and how to structure an evaluation that informs a decision.
- 11 Signed Mandates and Agent Accountability Why logs written by the agent cannot settle a dispute about what a user authorised, and how cryptographically signed mandates plus a hash-chained action record turn "the agent did it" into evidence someone can be held to.
- 12 Threat Modelling an LLM System The trust boundaries specific to model-based applications, an inventory of assets and adversaries worth enumerating, and how to turn that into controls rather than a document.