LLM Application Architecture
Routing, fallbacks, caching layers, structured state, and the reference shapes production LLM apps take.
12concepts
108flashcards
88minutes of reading
- 01 Batch and Asynchronous LLM Workloads How provider batch APIs trade a day of latency for half the price, how to shape traffic under token-bucket rate limits when you cannot wait, and why every asynchronous LLM job needs idempotent writes and a reconciliation step.
- 02 Conversation Branching and the Message Graph Why editing a turn or regenerating a reply turns a conversation into a tree rather than a list, what a branch invalidates downstream, and the storage and accounting shapes that follow.
- 03 Document Ingestion and Parsing Pipelines Why turning PDFs into text is the silent ceiling on retrieval quality, how layout-model pipelines and vision-language parsers differ in cost and failure, and how to re-ingest a changing corpus without reprocessing all of it.
- 04 Fallbacks, Timeouts and Degradation How to build an LLM feature that survives a provider outage, why the usual retry patterns need adjusting for generation, and what degrading gracefully looks like when the core capability is unavailable.
- 05 Model Routing and Cascades Why sending every request to the strongest model is usually wrong, the two routing patterns and what each requires, and the economics that decide whether routing pays.
- 06 The Tool Layer as a Dependency Tier How to budget timeouts across the tools inside a single model turn, why a tool failure belongs in the conversation rather than in an exception, and what tool definitions and results cost in context and cache.