LLM Application Architecture
Routing, fallbacks, caching layers, structured state, and the reference shapes production LLM apps take.
5concepts
60flashcards
35minutes of reading
- 01 Fallbacks, Timeouts and Degradation How to build an LLM feature that survives a provider outage, why the usual retry patterns need adjusting for generation, and what degrading gracefully looks like when the core capability is unavailable.
- 02 Model Routing and Cascades Why sending every request to the strongest model is usually wrong, the two routing patterns and what each requires, and the economics that decide whether routing pays.
- 03 Caching Layers for LLM Applications The four distinct caches in a mature LLM system, what each requires to be correct, and why semantic caching is the one that is dangerous.
- 04 Structured Output and Constrained Generation Why parsing free text is the wrong integration point, how constrained decoding guarantees valid syntax, and what a guarantee about form does not give you about content.
- 05 Where State Lives in an LLM Application Why the context window is the worst place to keep state, the four stores a mature application actually uses, and the assembly step that decides what the model sees.