Serving Systems
Prompt caching, gateways and routing, token accounting, and multi-tenant isolation.
13concepts
195flashcards
108minutes of reading
- 01 Autoscaling and Cold Starts in LLM Serving Why GPU utilisation is a useless autoscaling signal for LLM servers, what a cold start actually costs, and how to scale a fleet whose new replicas take minutes to become useful.
- 02 LLM Gateways and Routing Why every serious LLM deployment ends up behind a gateway, and how to choose between LiteLLM, Portkey, OpenRouter, and rolling your own.
- 03 Model Rollouts: Shadow, Canary, Rollback Why swapping the model behind a product is nothing like deploying code, how shadow traffic and canary stages catch what offline evals miss, and what has to be versioned together for rollback to mean anything.
- 04 Prompt Caching Infrastructure How Anthropic, OpenAI, and vLLM let you reuse the KV cache of repeated prefixes, what the cache key actually is, and the patterns that turn cache hit rate into a real bill reduction.
- 05 Semantic Caching and Response Reuse Returning a stored answer when a new prompt is merely similar to an old one converts an inference call into a vector lookup, and turns a correctness question into a similarity-threshold setting.
- 06 Serving SLOs: TTFT, TPOT and Goodput Why tokens per second is the wrong number to optimise, how TTFT and TPOT split the latency budget, and what goodput measures that throughput hides.
- 07 Token Accounting, Billing, and Quotas Why a single token counter is not enough, how to attribute spend across users and features without losing your mind, and the patterns that prevent one bad actor from spending the whole month's budget on a Tuesday afternoon.