LLM Application Architecture
Routing, fallbacks, caching layers, structured state, and the reference shapes production LLM apps take.
12concepts
108flashcards
88minutes of reading
- 01 Batch and Asynchronous LLM Workloads How provider batch APIs trade a day of latency for half the price, how to shape traffic under token-bucket rate limits when you cannot wait, and why every asynchronous LLM job needs idempotent writes and a reconciliation step.
- 02 Conversation Branching and the Message Graph Why editing a turn or regenerating a reply turns a conversation into a tree rather than a list, what a branch invalidates downstream, and the storage and accounting shapes that follow.
- 03 Document Ingestion and Parsing Pipelines Why turning PDFs into text is the silent ceiling on retrieval quality, how layout-model pipelines and vision-language parsers differ in cost and failure, and how to re-ingest a changing corpus without reprocessing all of it.
- 04 Fallbacks, Timeouts and Degradation How to build an LLM feature that survives a provider outage, why the usual retry patterns need adjusting for generation, and what degrading gracefully looks like when the core capability is unavailable.
- 05 Model Routing and Cascades Why sending every request to the strongest model is usually wrong, the two routing patterns and what each requires, and the economics that decide whether routing pays.
- 06 The Tool Layer as a Dependency Tier How to budget timeouts across the tools inside a single model turn, why a tool failure belongs in the conversation rather than in an exception, and what tool definitions and results cost in context and cache.
- 07 Caching Layers for LLM Applications The four distinct caches in a mature LLM system, what each requires to be correct, and why semantic caching is the one that is dangerous.
- 08 Permission-Aware Retrieval Why authorisation in a retrieval system has to happen inside the query rather than after it, how group-based ACLs flatten badly into index filters, and the leak that no retrieval filter can close.
- 09 Streaming Transport, Cancellation and Resumability Why an HTTP 200 tells you nothing about a streamed response, what the infrastructure between your server and the user does to a long-lived stream, and how to make a stream survive the connection that carried it.
- 10 Structured Output and Constrained Generation Why parsing free text is the wrong integration point, how constrained decoding guarantees valid syntax, and what a guarantee about form does not give you about content.
- 11 Tenant Isolation in LLM Applications Why a multi-tenant LLM application leaks through its caches, queues and memory rather than its database, how provider isolation units constrain your tenancy model, and what a shared prompt cache reveals through timing alone.
- 12 Where State Lives in an LLM Application Why the context window is the worst place to keep state, the four stores a mature application actually uses, and the assembly step that decides what the model sees.