Serving Systems
Prompt caching, gateways and routing, token accounting, and multi-tenant isolation.
13concepts
195flashcards
108minutes of reading
- 01 Autoscaling and Cold Starts in LLM Serving Why GPU utilisation is a useless autoscaling signal for LLM servers, what a cold start actually costs, and how to scale a fleet whose new replicas take minutes to become useful.
- 02 LLM Gateways and Routing Why every serious LLM deployment ends up behind a gateway, and how to choose between LiteLLM, Portkey, OpenRouter, and rolling your own.
- 03 Model Rollouts: Shadow, Canary, Rollback Why swapping the model behind a product is nothing like deploying code, how shadow traffic and canary stages catch what offline evals miss, and what has to be versioned together for rollback to mean anything.
- 04 Prompt Caching Infrastructure How Anthropic, OpenAI, and vLLM let you reuse the KV cache of repeated prefixes, what the cache key actually is, and the patterns that turn cache hit rate into a real bill reduction.
- 05 Semantic Caching and Response Reuse Returning a stored answer when a new prompt is merely similar to an old one converts an inference call into a vector lookup, and turns a correctness question into a similarity-threshold setting.
- 06 Serving SLOs: TTFT, TPOT and Goodput Why tokens per second is the wrong number to optimise, how TTFT and TPOT split the latency budget, and what goodput measures that throughput hides.
- 07 Token Accounting, Billing, and Quotas Why a single token counter is not enough, how to attribute spend across users and features without losing your mind, and the patterns that prevent one bad actor from spending the whole month's budget on a Tuesday afternoon.
- 08 Admission Control and Load Shedding Why an overloaded LLM server degrades for everyone at once, how token-aware admission control keeps latency SLOs intact by rejecting work early, and what graceful degradation looks like when you would rather not say no.
- 09 Disaggregated Prefill and Decode Serving Why prefill and decode want opposite hardware and parallelism, how splitting them across separate GPU pools raises goodput, and what the KV cache transfer costs.
- 10 Multi-Tenant Serving and Isolation Serving many tenants from one model is cheap and easy; giving each tenant their own fine-tune is expensive and hard. S-LoRA and per-request LoRA serving collapse the trade-off, but only for tenants who can share a base model.
- 11 Prefix-Aware Routing and KV Cache Reuse Why load-balancing LLM requests round-robin throws away computed KV cache, and how routing on prompt prefix turns a fleet's caches into a shared asset.
- 12 Queueing Theory for LLM Serving Little's Law fixes the maximum request rate a batched LLM server can sustain at a given latency, and the heavy-tailed distribution of output lengths explains why queues form long before the GPU is saturated.
- 13 Request Scheduling and Preemption Shortest-job-first minimises average waiting time and requires knowing job length, which an LLM server cannot know; the workarounds are predicting the rank of output lengths or preempting at token granularity.