Goodput, Not Throughput: The Metric That Decides Whether Your LLM Deployment Works

The benchmark said 4,200 tokens per second. The support ticket said "it hangs for five seconds and then dumps a wall of text."

Both were accurate. The team had tuned their deployment for aggregate throughput, which meant large batches and an admission policy that let long prompts in whenever a slot opened. Total token rate went up and stayed up. What the number could not show was that a 16,000-token prefill seizes the GPU for hundreds of milliseconds at a time, so every request already streaming stops mid-sentence, and every new request waits behind whatever queue has formed. Median time to first token was 1.2 seconds. The 99^th percentile was over six.

Nothing was broken. They were optimising a number that does not describe the experience of using the service, and it is the number nearly every inference benchmark reports.

Why this matters: Almost every serving optimisation of the last three years, continuous batching, chunked prefill, disaggregation, prefix caching, speculative decoding, either helps or hurts depending on which latency constraint binds. Without an SLO-aware metric you cannot tell which, and the same change that looks like a 30% win in tokens per second can be a regression for users.

TL;DR

• A request is two workloads with opposite physics: prefill is compute-bound and scales with prompt length; decode is memory-bandwidth-bound and roughly independent of it.
• That split gives two user-facing numbers, TTFT and TPOT, that respond to different knobs in opposite directions. Total latency is \(\text{TTFT} + \text{TPOT} \times (N_{\text{out}} - 1)\).
• Goodput is the request rate sustained while meeting both constraints. DistServe made it the objective function rather than a reporting convention (Zhong et al., OSDI 2024, arXiv:2401.09670).
• Chunked prefill can score as a throughput regression and still be a large goodput win, because slicing a prefill costs efficiency and buys latency (Agrawal et al., OSDI 2024, arXiv:2403.02310).
• Disaggregating prefill and decode onto separate GPU pools reported 7.4x more requests, or a 12.6x tighter SLO, at over 90% SLO attainment. The price is a KV cache transfer on every request's critical path.
• An LLM replica is not stateless. Round-robin load balancing across replicas throws away computed KV cache, and prefix-aware routing recovers it.
• GPU utilisation is useless as an autoscaling signal because continuous batching pins it near 100% by design. Queue depth and KV cache occupancy move first.
• Speculative decoding cuts TPOT at low load and can reduce goodput under saturation, because rejected drafts consume batch slots real requests needed.

At a Glance

flowchart LR
    R["Request<br/>prompt + SLO"] --> Q["Queue<br/>admission control"]
    Q --> P["Prefill<br/>compute-bound"]
    P --> K["KV cache<br/>paged blocks"]
    K --> D["Decode loop<br/>bandwidth-bound"]
    D --> S["Stream tokens"]
    K -.->|"prefix reuse"| P
    P -->|"TTFT"| M["SLO check"]
    D -->|"TPOT"| M
    M --> G["Goodput:<br/>requests meeting both"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

    class R,Q blue
    class P purple
    class K amber
    class D,S teal
    class M,G emerald

Before Continuous Batching

Serving LLMs began as an adaptation of ordinary model serving, and every step since has been the removal of an assumption that turned out to be false.

timeline
    title How LLM serving learned that a request is two workloads
    2020 : Request-level batching
         : A batch finishes when its slowest member finishes; short requests wait
    2022 : Orca introduces iteration-level scheduling
         : Continuous batching reports 36.9x throughput over FasterTransformer at equal latency
    2023 : vLLM and PagedAttention
         : KV cache managed like virtual memory, 2-4x throughput over prior systems
    2023 : SGLang RadixAttention
         : Prefix cache in a radix tree, automatic reuse across requests
    2024 : DistServe and Sarathi-Serve
         : Goodput becomes the objective; prefill and decode are separated or interleaved
    2024 : Mooncake goes to production at Kimi scale
         : KVCache-centric cluster architecture, best paper at FAST 2025
    2025 : Dynamo, llm-d, LMCache
         : KV-cache-aware routing and tiered cache become standard infrastructure

The first assumption to fall was that a batch is a unit of work. Under request-level batching, a batch is admitted, runs to completion, and retires together, so a request generating 20 tokens sits in the GPU until the request generating 800 tokens finishes. Orca replaced it with iteration-level scheduling, where the scheduler makes a decision every single decoding step and a finished request leaves its slot immediately, reporting a 36.9x throughput improvement over FasterTransformer at the same latency on GPT-3 175B (Yu et al., OSDI 2022). This is continuous batching, and every serving system now does it.

The second was that KV cache should be a contiguous allocation. Reserving the maximum possible sequence length per request wastes most of it, and fragmentation wastes more. PagedAttention borrowed virtual memory: KV cache lives in fixed-size blocks with an indirection table, so allocation is on demand and blocks can be shared across sequences. vLLM reported 2-4x throughput over the previous state of the art at the same latency (Kwon et al., SOSP 2023, arXiv:2309.06180).

Both advances raised throughput, and both left the deeper assumption standing: that prefill and decode are the same kind of work and belong on the same GPU in the same iteration. That is the assumption 2024 spent its time removing.

[IMAGE: Gantt-style timeline of four requests under request-level batching versus continuous batching. In the first, all four occupy slots until the longest finishes; in the second, finished requests retire and new ones enter mid-flight. Caption: "Iteration-level scheduling turns a batch from a unit of work into a rolling population."]

How Latency Actually Decomposes

Two phases, opposite physics

Prefill processes the entire prompt in one pass. Thousands of tokens flow through large matrix multiplications, the arithmetic units saturate, and the phase is compute-bound. Its cost scales with prompt length, roughly linearly in tokens for the feed-forward work and quadratically in the attention term before FlashAttention-style kernels, linearly in memory traffic after.

Decode generates one token at a time. Each step multiplies a single token's activations against the full weight matrix and attends over the whole KV cache. The arithmetic intensity is terrible: for a batch of one, the GPU reads every parameter from HBM to perform a handful of FLOPs per byte, so the phase is bandwidth-bound and the arithmetic units idle. Increasing batch size amortises the same weight read across more sequences, which is why decode throughput improves with batching and prefill throughput barely does.

A useful way to hold it: prefill is a matrix-matrix product, decode is a matrix-vector product repeated. Everything else follows from that.

[IMAGE: Roofline-style plot with arithmetic intensity on the x-axis and achievable FLOPs on the y-axis, with prefill plotted far right under the compute-bound roof and decode plotted far left on the bandwidth-bound slope, for batch sizes 1, 8 and 64. Show decode migrating rightward as batch size grows. Caption: "Prefill and decode sit on opposite sides of the ridge point. Batching is the only thing that moves decode toward the compute roof."]

The two numbers users feel

TTFT is queueing plus scheduling plus the prefill pass. It scales with prompt length and with how loaded the queue is. Users experience it as "did it hear me".

TPOT, sometimes called inter-token latency, is the steady-state gap between output tokens. It scales with batch size and memory bandwidth and is essentially independent of prompt length. Users experience it as reading speed.

For a 500-token answer the second term dominates by an order of magnitude. For code completion the first term is nearly the whole experience. Comfortable reading sits around 30 to 50 ms per token, so a TPOT above roughly 80 ms feels laboured no matter how fast the first token arrived.

The fourth row changed the shape of the field. An agent making a dozen tool calls pays TTFT a dozen times, so latency that was acceptable in chat becomes the dominant cost of an agent run. This is why prefix caching stopped being an optimisation and became an architectural requirement.

Goodput

Goodput is the request rate a system sustains while meeting both its TTFT and TPOT targets. Serving 100 requests per second with 40% of them violating TTFT is a goodput of 60. DistServe defined it as the maximum rate servable within both constraints per GPU and optimised against it directly (Zhong et al., OSDI 2024).

The reframing exposes a tradeoff that throughput accounting erases: the two phases compete for the same GPU. Batch more requests and throughput rises while TPOT worsens for everyone in the batch. Admit a long prompt and its prefill blocks the decode loop, spiking TPOT for requests already streaming. Sarathi-Serve named this the throughput-latency tradeoff and attacked it with chunked prefill, slicing long prefills into pieces small enough to co-schedule with decodes so no decode iteration stalls, reporting up to 2.6x higher serving capacity for Mistral-7B on one A100 and 3.7x for Yi-34B on two (Agrawal et al., OSDI 2024, arXiv:2403.02310).

Read that as a goodput result. Measured as raw tokens per second, chunked prefill can look like a small regression, because slicing a prefill is marginally less efficient than running it whole. It is a clear win that the naive metric scores as a loss.

Seeing It in Motion

Interference in a colocated scheduler

sequenceDiagram
    participant A as Streaming request A
    participant B as Streaming request B
    participant S as Scheduler
    participant C as New request C (16k prompt)

    A->>S: decode step (8 ms)
    B->>S: decode step (batched, same 8 ms)
    C->>S: arrives, needs prefill
    Note over S: colocated: prefill runs as one iteration
    S->>S: prefill 16k tokens (~600 ms)
    Note over A,B: A and B produce no tokens for 600 ms
    S-->>C: first token
    A->>S: decode resumes
    B->>S: decode resumes

The three responses to that stall

flowchart TB
    subgraph Colo["Colocated (baseline)"]
        direction TB
        X1["One GPU pool"] --> X2["Prefill blocks decode"]
        X2 --> X3["TPOT spikes on every long prompt"]
    end
    subgraph Chunk["Chunked prefill"]
        direction TB
        Y1["One GPU pool"] --> Y2["Prefill sliced into chunks"]
        Y2 --> Y3["Chunks ride with decodes,<br/>no stall"]
    end
    subgraph Disagg["Disaggregated"]
        direction TB
        Z1["Prefill pool"] --> Z2["KV transfer"]
        Z2 --> Z3["Decode pool"]
        Z3 --> Z4["Zero interference,<br/>independent scaling"]
    end

    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class X1,X2,X3 rose
    class Y1,Y2,Y3 amber
    class Z1,Z3,Z4 emerald
    class Z2 slate

KV cache as fleet state

stateDiagram-v2
    [*] --> Computed: prefill produces blocks
    Computed --> Active: sequence decoding
    Active --> Cached: request finishes, blocks retained
    Cached --> Reused: new request shares the prefix
    Reused --> Active
    Cached --> Evicted: KV memory pressure
    Evicted --> Offloaded: tiered to CPU DRAM or NVMe
    Offloaded --> Reused: reload beats recompute
    Evicted --> [*]: recompute next time

Watch It Run

The companion diagram (llm-serving-goodput-not-throughput.drawio) animates a request through a disaggregated deployment: the router scoring replicas on cache overlap and load, prefill on one pool, the KV transfer, the decode loop iterating token by token, and the SLO monitor feeding back into the autoscaler.

By the Numbers

Sources: Orca (Yu et al., OSDI 2022), vLLM (Kwon et al., SOSP 2023, arXiv:2309.06180), Sarathi-Serve (Agrawal et al., OSDI 2024, arXiv:2403.02310), DistServe (Zhong et al., OSDI 2024, arXiv:2401.09670), Mooncake (Qin et al., FAST 2025, arXiv:2407.00079). Each number comes from its authors' own evaluation on their chosen hardware and workload mix, so the rows are not comparable to each other; read them as evidence that the phase-separation idea pays, not as a ranking.

Two quantities you can derive yourself, which matter more for capacity planning than any published speedup:

KV cache per token. For a model with \(L\) layers, \(H\) key-value heads and head dimension \(d_h\), in 2-byte precision:

The leading 2 is key plus value, the second is bytes per element. A 70B-class model with 80 layers, 8 KV heads (grouped-query attention) and head dimension 128 gives \(2 \times 2 \times 80 \times 8 \times 128 = 327{,}680\) bytes, about 320 KB per token, so a 16,000-token prompt holds roughly 5 GB of KV cache. That single number explains why long-context serving is a memory problem, why disaggregation's transfer cost is measured in gigabytes, and why grouped-query attention was adopted so quickly.

Cold start. A 70B model in BF16 is about 140 GB of weights. Pulling that from object storage over a 10 Gbps link is roughly 112 seconds at line rate, before host-to-device copy, tensor-parallel sharding, kernel warm-up and CUDA graph capture. This is why a new replica is two to ten minutes from useful, and why autoscaling on a lagging signal cannot work.

[IMAGE: Two-axis plot with request rate on the x-axis and two curves: raw throughput in tokens per second, rising and then plateauing, and goodput under a 500 ms TTFT / 50 ms TPOT SLO, rising and then collapsing past the knee. Annotate the gap between the plateau and the collapse. Caption: "Past the knee, throughput keeps looking healthy while goodput falls off a cliff. Only one of these curves matches what users report."]

A Concrete Example

A deployment serves a documentation assistant on 8 GPUs. The system prompt with tool definitions is 2,000 tokens, retrieved context averages 6,000 tokens, and the user turn is around 200. Answers average 400 tokens. The SLO is TTFT under 800 ms and TPOT under 50 ms.

Baseline, colocated, round-robin routing. Every request prefills roughly 8,200 tokens. At an assumed 6,000 prompt-tokens per second per GPU for prefill on this model and hardware, that is about 1.37 seconds of prefill work per request. Prefill alone breaches the TTFT budget before any queueing. Meanwhile each long prefill stalls the decode loop, so TPOT spikes above 50 ms whenever one lands. Measured goodput: near zero at any meaningful load, despite a healthy-looking tokens-per-second figure driven by the decode phase.

Change 1: prefix caching plus prefix-aware routing. The first 2,000 tokens are identical for every request. Retrieved context varies, so the shared prefix ends at token 2,000. With automatic prefix caching, a cache hit skips prefill for that span, cutting prefill work per request from 8,200 to 6,200 tokens, about 24%. TTFT falls from ~1.37 s to ~1.03 s. Still breaching, but the routing change matters more than the arithmetic suggests: without prefix-aware routing the hit rate across 8 replicas would have been about \(1/8\), so the effective saving would have been closer to 3%.

Change 2: move the volatile part. The team discovers a Request-ID interpolated into the system prompt's second line. Moving it to the end of the prompt extends the shared prefix from 2,000 tokens to the full system block. No other change. This costs nothing and is the highest-return edit in the entire exercise.

Change 3: chunked prefill. Prefills are sliced into 512-token chunks that co-schedule with decodes. TPOT stops spiking: measured p99 TPOT drops from 140 ms to 46 ms, inside the SLO. TTFT is roughly unchanged, perhaps 3% worse from slicing overhead. Goodput rises sharply because the TPOT constraint stopped failing.

Change 4: disaggregation. Prefill moves to 3 GPUs with tensor parallelism cutting its latency, decode to 5 GPUs running large batches. TTFT falls under the 800 ms budget. The KV transfer for a 6,200-token uncached remainder at 320 KB per token is about 2 GB, which over NVLink at hundreds of GB/s is a few milliseconds and overlaps with the prefill tail. Over 10 Gbps Ethernet it would be roughly 1.6 seconds and would have destroyed the gain outright.

The final configuration serves within SLO. Three of the four changes were scheduling and placement decisions, one was a prompt edit, and none involved a faster model or better hardware.

[IMAGE: Stacked bar chart showing TTFT decomposition (queue wait, cache-miss prefill, cached prefix skipped, transfer) across the five configurations in the worked example, with a horizontal line at the 800 ms SLO. Caption: "Each intervention removes a different component of TTFT. Only the last one requires new infrastructure."]

Where It Breaks

The KV transfer can cost more than the interference it removed

Disaggregation moves gigabytes per request between pools. Over NVLink or InfiniBand with RDMA the transfer overlaps with the tail of prefill and largely disappears. Over ordinary Ethernet it becomes the dominant term. This is why the industry effort went into transfer layers rather than the architecture itself: NVIDIA's NIXL inside Dynamo, adopted by llm-d, TensorRT-LLM, SGLang and vLLM, and Mooncake's KVCache-centric design that pools idle CPU DRAM and SSD across the cluster as a shared cache tier.

Disaggregation also loses below roughly a handful of GPUs. Splitting a small fleet leaves each pool too small to batch well, and the static partition wastes capacity that colocation would have shared. Chunked prefill is the correct answer at that scale.

Anything variable at the front of the prompt destroys cache reuse

Prefix caching matches on a shared leading token sequence. A timestamp, a request id, or a user's name interpolated into the system prompt moves the divergence point to nearly token zero, and every request pays full prefill. The fix is free and almost never applied on the first try: volatile content goes at the end, after the stable system instructions and tool definitions.

[IMAGE: Two prompt layouts drawn as horizontal token bars with the shared-prefix region shaded green and the divergent region shaded red. Top: a request ID placed in line two of the system prompt, leaving almost the entire bar red. Bottom: the same ID moved after the retrieved context, leaving 8,000 tokens green. Caption: "Same prompt, same tokens, same model. The only difference is where the volatile field sits."]

Cache and concurrency compete for the same memory

Blocks retained for future reuse are blocks unavailable to active sequences. A more aggressive prefix cache raises hit rate and lowers maximum batch size, and the crossover depends on how repetitive the traffic is. There is no default that is right for both a high-share agent workload and diverse one-shot traffic.

Content-hashed cache sharing is a timing side channel

If KV blocks are shared across tenants by content hash, an unusually fast TTFT reveals that someone else recently submitted the same prefix, which can confirm the presence of a specific document or prompt. Where that matters, partition the cache namespace per tenant and accept the lower hit rate.

Autoscaling on a metric that never moves

Continuous batching is designed to keep the GPU busy, so utilisation pins near 100% whether the system is comfortable or drowning. Scaling on it is scaling on noise. The signals that move first are pending queue depth, KV cache occupancy, and preemption rate; vLLM exports num_requests_waiting and gpu_cache_usage_perc for exactly this reason. TTFT p95 against SLO is worth alerting on and useless for scaling on, because by the time it breaches, the replica you now request arrives minutes after the spike.

That asymmetry sets the policy: scale up fast on leading signals, scale down slowly with a generous stabilisation window, since releasing a replica saves one GPU-minute and re-acquiring it costs several minutes of cold start. And drain properly, because a replica marked for termination may still be generating for another minute; kill it and those requests fail mid-stream.

Speculative decoding can reduce goodput

It cuts TPOT beautifully at low load by verifying several draft tokens per step. Under saturation, rejected drafts consume compute and batch slots that queued requests needed, so it lowers goodput while each individual request still looks faster in isolation. It is a latency optimisation purchased with spare capacity, which means it pays exactly when capacity is spare.

Alternative Designs

These compose rather than compete. A serious 2026 deployment runs continuous batching with paged KV, chunked prefill or disaggregation depending on fleet size, prefix caching with a cache-aware router, and speculative decoding gated on current load.

How It Is Used in Practice

The reference stack has converged. vLLM or SGLang as the engine, giving paged KV, continuous batching and automatic prefix caching. A control plane above it, Dynamo or llm-d, providing KV-cache-aware routing and, at scale, disaggregated pools connected by NIXL. A tiered cache such as LMCache or Mooncake's store, treating KV blocks as objects that outlive the request that produced them. Kubernetes HPA scaling on queue depth rather than utilisation.

The routing objective in these systems pairs two terms:

Overlap alone creates a hotspot, since the most popular prefix pulls all its traffic to one worker. Load alone is round-robin and discards the cache. Dynamo's KV router scores workers on exactly this pairing, weighing prefill cost from newly computed blocks against decode cost from active blocks.

Benchmarking practice lags behind the systems. To produce a number that means something: state the input and output length distributions, measure under offered load rather than one request at a time, report percentiles rather than means, and report goodput against a stated SLO pair. Any throughput number without a latency constraint attached can be improved by making the service worse, which is precisely how the deployment at the top of this article was tuned.

[IMAGE: System architecture diagram of a production LLM serving stack: gateway, KV-cache-aware router, prefill pool and decode pool connected by an RDMA transfer layer, a tiered KV store spanning HBM/DRAM/NVMe, and a control plane consuming queue-depth and SLO-attainment metrics to drive autoscaling. Caption: "The 2026 reference stack. Three of its five components exist because prefill and decode want different things."]

Insights Worth Remembering

1. Throughput is a server metric; latency is a request metric. Goodput is the only one of the three that describes the product. Optimising the first alone reliably produces a system that is fast on paper and slow to use.

2. Prefill and decode are not two views of one workload. One is a matrix-matrix product and compute-bound; the other is a matrix-vector product and bandwidth-bound. Nearly every serving advance since 2022 is a consequence of taking that difference seriously.

3. A change can improve latency and reduce goodput. Speculative decoding under saturation is the cleanest example. Per-request improvements measured in isolation do not aggregate.

4. An LLM replica is stateful and your load balancer does not know it. The KV cache is fleet state. Round-robin across \(N\) replicas gives a \(1/N\) chance of a warm prefix and makes every replica cache the same thing.

5. The highest-return serving optimisation is often a prompt edit. Moving a timestamp from the top of the system prompt to the bottom can restore an entire prefix cache. It costs nothing and ships in an afternoon.

6. GPU utilisation is a lie by design. Continuous batching exists to keep it at 100%. Any autoscaling policy keyed on it is keyed on a constant.

7. KV cache per token is the number to memorise for your model. It determines maximum concurrency, long-context viability, disaggregation transfer cost, and how much cache you can retain. Everything downstream is arithmetic on it.

8. Cold start makes autoscaling a forecasting problem, not a reactive one. Two to ten minutes to a useful replica means you scale on what predicts load, not on what reports it.

9. Chunked prefill and disaggregation are not rivals. They answer the same question at different fleet sizes, and the crossover is roughly where a split pool can still batch well.

10. A benchmark without an SLO is a marketing artifact. Stating the latency constraints is what makes a serving number falsifiable.

Open Questions

Where exactly is the disaggregation crossover? The published gains come from large fleets with fast fabrics. The fleet size and interconnect bandwidth at which disaggregation overtakes chunked prefill is workload-dependent and, as of early 2026, not characterised in a way practitioners can apply without measuring it themselves.

Should the prefill/decode split be dynamic? A fixed partition is wrong the moment traffic mix shifts, and several systems now rebalance pools at runtime. Whether the control-plane complexity pays for itself outside hyperscale deployments is unresolved.

How far does KV cache tiering go? Mooncake and LMCache treat KV blocks as first-class storage objects across DRAM and SSD. The open question is whether a cluster-wide KV cache becomes a shared service, like a CDN for prefixes, or stays a per-deployment optimisation. Evidence so far is promising in long-context regimes and unproven for diverse short-prompt traffic.

Can SLO-aware scheduling be learned rather than tuned? Admission control, chunk size, batch composition and speculation depth are currently hand-tuned. Framing them as one policy optimised against goodput is an obvious formulation, and there is no deployed system demonstrating it beats good heuristics.

Does speculative decoding survive at scale? Its benefit is inversely related to load, which is the opposite of what a capacity planner wants. Load-adaptive speculation, turning it off as the queue grows, is straightforward in principle; whether the switching costs and complexity are worth it is being worked out now.

Sources and Further Reading

Foundational systems papers

1. Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., & Chun, B.-G. (2022). "Orca: A Distributed Serving System for Transformer-Based Generative Models." OSDI 2022. USENIX

2. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). "Efficient Memory Management for Large Language Model Serving with PagedAttention." SOSP 2023. arXiv:2309.06180

3. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., & Zhang, H. (2024). "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving." OSDI 2024. arXiv:2401.09670

4. Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., Tumanov, A., & Ramjee, R. (2024). "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve." OSDI 2024. arXiv:2403.02310

Cache-centric architectures

5. Qin, R., Li, Z., He, W., Zhang, M., Wu, Y., Zheng, W., & Xu, X. (2024). "Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving." FAST 2025 (Best Paper). arXiv:2407.00079

6. Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., & Sheng, Y. (2023). "SGLang: Efficient Execution of Structured Language Model Programs." arXiv:2312.07104

Documentation and engineering references

7. NVIDIA. "KV Cache Aware Routing." NVIDIA Dynamo Documentation. docs.nvidia.com

8. NVIDIA. (2025). "How NVIDIA Dynamo Accelerates llm-d Community Initiatives for Advancing Large-Scale Distributed Inference." NVIDIA Technical Blog

9. vLLM project. "Production Metrics." docs.vllm.ai

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.