Performance and Soak Gates Before Rollout

A new serving build passes every quality gate, then a ten-minute load test at a nominal 200 requests per second reports a p99 of 310 ms, comfortably inside a 500 ms budget. It is promoted on Monday and pages at 02:00 on Thursday with p99 above four seconds. Nothing regressed between Monday and Thursday. The load test measured the wrong quantity, and the problem it would have caught needed six hours to appear.

A closed loop cannot measure a queue

Schroeder, Wierman and Harchol-Balter showed that open and closed workload generators produce substantially different system behaviour, and that designers routinely fail to notice which one their harness implements (Schroeder et al., 2006, Open Versus Closed: A Cautionary Tale, NSDI '06). A closed generator holds \(N\) virtual users, each of which sends its next request only after the previous response arrives. When the server stalls, every user stalls with it, and the requests that a real arrival process would have queued are simply never sent.

Gil Tene named the resulting error coordinated omission: the harness records service time while the user experiences response time, and the stall vanishes from the histogram instead of dominating it (Tene, 2015, How NOT to Measure Latency, QCon San Francisco).

The arithmetic is unforgiving. By Little's law a closed harness with \(N = 50\) and a 250 ms mean response time offers

which is the number in the opening. Let the service degrade to a 500 ms mean and the same harness offers 100 req/s. The load fell by half in response to the degradation, the queue never built, and the report says the service handled 200 req/s with a 310 ms p99. Fix it by driving a fixed arrival rate, timestamping each request at its scheduled send time rather than its actual send time, recording into an HdrHistogram-style structure, and reporting the maximum alongside the percentiles.

What an LLM service needs measured

One latency number is not enough. Time to first token and time per output token have different causes, different fixes and different user consequences, so gate them separately and gate on goodput, the fraction of requests meeting both targets at once (see serving SLOs). The input and output length distributions have to match production rather than its mean, because the tail of the latency distribution is produced by the tail of the prompt length distribution, and a synthetic workload of uniform 512-token prompts cannot produce it.

What only time reveals

Memory growth is the classic one and remains the most common. Allocator behaviour is the specifically modern one: PagedAttention exists because contiguous KV cache allocation wastes memory through fragmentation and duplication, which caps the achievable batch size, and the paged design reports 2 to 4 times the throughput at the same latency (Kwon et al., 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP, arXiv:2309.06180). Fragmentation is a function of the sequence-length mix accumulated over hours, which is exactly what a ten-minute run with a narrow length distribution will not produce.

The rest of the list is unglamorous and worth asserting on anyway: file descriptors and connection pools under churn, log volume filling a disk, a semantic cache that is 95 percent hit at minute five and 40 percent at hour six once the key space turns over, and the hourly batch job that collides with serving on the same node.

A threshold that is not noise

Two runs of an unchanged service disagree. Run the baseline twice on the same hardware pool, take the run-to-run spread of the metric you intend to gate, and set the threshold at about three times that spread. A gate set at a round number instead measures the variance of your staging environment, fails for reasons nobody can reproduce, and gets disabled. Where the runner pool is heterogeneous, gate on the paired difference against a baseline run in the same window, not on an absolute value.

When it breaks

The generator saturates before the service does. A single-process harness doing TLS and JSON work competes for the same CPU as nothing useful. Check that the generator's own CPU is under half, or that adding a second generator host changes the result.

Warm-up contaminates everything. First-request compilation, CUDA graph capture and autotuning take seconds. Run a discarded warm-up phase and start the histogram after it, or the p99 is a measurement of initialisation.

Staging is shared. Another team's job on the same node produces a regression in your numbers that does not exist in your code, and the only defence is recording node-level utilisation alongside the latency so the artefact is visible afterwards.

Cache state is never stated. A fully warm run and a fully cold run bracket production and neither equals it. Gate on a mix with a declared hit rate, and record what it was.

Soak costs real hours. A six-hour soak on production-shaped hardware is a real bill and does not belong on the pull request path. Weekly on the main branch is a defensible compromise, and it is only worth anything if someone reads the result.