LLM Application Architecture
Routing, fallbacks, caching layers, structured state, and the reference shapes production LLM apps take.
12concepts
108flashcards
88minutes of reading
- 01 Caching Layers for LLM Applications The four distinct caches in a mature LLM system, what each requires to be correct, and why semantic caching is the one that is dangerous.
- 02 Permission-Aware Retrieval Why authorisation in a retrieval system has to happen inside the query rather than after it, how group-based ACLs flatten badly into index filters, and the leak that no retrieval filter can close.
- 03 Streaming Transport, Cancellation and Resumability Why an HTTP 200 tells you nothing about a streamed response, what the infrastructure between your server and the user does to a long-lived stream, and how to make a stream survive the connection that carried it.
- 04 Structured Output and Constrained Generation Why parsing free text is the wrong integration point, how constrained decoding guarantees valid syntax, and what a guarantee about form does not give you about content.
- 05 Tenant Isolation in LLM Applications Why a multi-tenant LLM application leaks through its caches, queues and memory rather than its database, how provider isolation units constrain your tenancy model, and what a shared prompt cache reveals through timing alone.
- 06 Where State Lives in an LLM Application Why the context window is the worst place to keep state, the four stores a mature application actually uses, and the assembly step that decides what the model sees.