Serving Systems
Prompt caching, gateways and routing, token accounting, and multi-tenant isolation.
8concepts
66flashcards
69minutes of reading
- 01 Disaggregated Prefill and Decode Serving Why prefill and decode want opposite hardware and parallelism, how splitting them across separate GPU pools raises goodput, and what the KV cache transfer costs.
- 02 Multi-Tenant Serving and Isolation Serving many tenants from one model is cheap and easy; giving each tenant their own fine-tune is expensive and hard. S-LoRA and per-request LoRA serving collapse the trade-off, but only for tenants who can share a base model.
- 03 Prefix-Aware Routing and KV Cache Reuse Why load-balancing LLM requests round-robin throws away computed KV cache, and how routing on prompt prefix turns a fleet's caches into a shared asset.