Serving Systems
Prompt caching, gateways and routing, token accounting, and multi-tenant isolation.
13concepts
195flashcards
108minutes of reading
- 01 Admission Control and Load Shedding Why an overloaded LLM server degrades for everyone at once, how token-aware admission control keeps latency SLOs intact by rejecting work early, and what graceful degradation looks like when you would rather not say no.
- 02 Disaggregated Prefill and Decode Serving Why prefill and decode want opposite hardware and parallelism, how splitting them across separate GPU pools raises goodput, and what the KV cache transfer costs.
- 03 Multi-Tenant Serving and Isolation Serving many tenants from one model is cheap and easy; giving each tenant their own fine-tune is expensive and hard. S-LoRA and per-request LoRA serving collapse the trade-off, but only for tenants who can share a base model.
- 04 Prefix-Aware Routing and KV Cache Reuse Why load-balancing LLM requests round-robin throws away computed KV cache, and how routing on prompt prefix turns a fleet's caches into a shared asset.
- 05 Queueing Theory for LLM Serving Little's Law fixes the maximum request rate a batched LLM server can sustain at a given latency, and the heavy-tailed distribution of output lengths explains why queues form long before the GPU is saturated.
- 06 Request Scheduling and Preemption Shortest-job-first minimises average waiting time and requires knowing job length, which an LLM server cannot know; the workarounds are predicting the rank of output lengths or preempting at token granularity.