A load generator reports a p99 of 30 ms at 5,000 rps. In production at the same rate, users report multi-second stalls, and the service's own latency metrics agree with the load test. Both measurements are honest. What is being missed?

The first three things to look at, in order

1. Is the generator closed-loop or open-loop? Closed-loop means a fixed number of virtual users, each sending its next request only after the previous one returns. Open-loop means requests are issued on a schedule regardless of responses.
2. Is latency measured from the intended send time or the actual send time?
3. The shape of the distribution, not its percentiles. A histogram with a suspicious gap where the slow samples should be is the signature.

The diagnosis: coordinated omission

In a closed-loop generator with N virtual users, when the server stalls for two seconds, every virtual user stalls with it. They do not send the requests they would have sent during those two seconds. When the server recovers, each user sends one request, which is served quickly, and records a fast sample.

So a two-second stall contributes one slow sample per virtual user, where a real population — which keeps arriving during the stall, because users do not coordinate with your server — would have contributed thousands. The generator has coordinated with the server's pauses and omitted precisely the requests that would have been slow.

The distortion is not marginal. A system that is healthy for 99% of a minute and stalled for 600 ms can report a single-digit-millisecond p99 while a substantial fraction of real users waited over a second.

Why the service's own metrics agree with the wrong number

They measure from the moment the request was accepted. A request sitting in the kernel accept backlog, or waiting on a connection the server has not yet accepted, does not exist from the server's point of view. Server-side latency measures service time; what the user feels is queueing time plus service time, and during a stall the first term is everything.

This is why "our metrics look fine" and "customers are complaining" are frequently both true.

The fix

• Generate open-loop. Issue requests at a fixed arrival rate on a schedule, regardless of whether earlier responses have returned. Backlog then accumulates on the client, which is the honest simulation of real arrivals.
• Record latency from the intended send time, so the samples a stall suppressed are reconstructed rather than lost. Good load tools offer this explicitly.
• Measure outside the process — at the load balancer and in the client — as the primary latency signal.
• Instrument the accept queue: connection establishment time and accept backlog depth are where the omitted latency lives.

What open-loop generation costs: the client now needs enough capacity to keep issuing while the server is stalled, so a load generator that fails under its own backlog produces a different lie. Size the generator fleet for the arrival rate, not for the response rate.

The alert that would have caught it earlier

Age of the oldest queued request, or accept backlog depth. Both exist before a request is served, and therefore before server-side latency exists at all. Any signal computed from served requests is structurally blind to requests that are still waiting.

When this is the wrong answer, and when not to blame the generator

If the stalls are visible in server-side latency and the load test still disagrees, the cause is workload realism instead: cache warmth, data cardinality, tenant mix, or a payload distribution the test does not reproduce. Coordinated omission is specifically the case where the server-side number is correct and the user-side number is much worse.