Coordinated Omission

A closed-loop load generator runs N virtual users, each sending its next request only after the previous response returns. It is the default model in most load-testing tools, it is simple to reason about, and it produces latency numbers that are systematically and severely optimistic.

When the server stalls for two seconds, every virtual user stalls with it. None of them issues the requests they would have issued during those two seconds. When the server recovers, each sends one request, is served quickly, and records a fast sample.

The stall therefore contributes one slow observation per virtual user, where a real population — which keeps arriving, because users do not coordinate with your server — would have contributed thousands. The generator has synchronised itself with the system's pauses and omitted precisely the measurements that mattered. Gil Tene named the effect and it has been part of the latency-measurement literature since around 2013.

Why it matters

It makes the tail, which is the only part of the distribution worth measuring, unmeasurable. A system healthy for 99% of a minute and stalled for 600 ms can report a single-digit-millisecond p99 while a large fraction of real users waited over a second. The number is not slightly wrong; it is wrong by orders of magnitude in the region where decisions are made.

It also explains the most common disagreement in production: server-side metrics look fine and customers report stalls. Both are honest. Server-side latency measures service time and begins when the request is accepted; a request waiting in the accept backlog does not exist yet from the server's point of view.

Implementation patterns

• Generate open-loop. Issue requests on a fixed arrival schedule regardless of whether earlier responses returned, so backlog accumulates on the client as it would in reality.
• Record latency from the intended send time, not the actual one, which reconstructs the samples a stall would otherwise suppress. Tools that support this usually call it exactly that.
• Size the generator for the arrival rate, not the response rate, or the generator saturates under its own backlog and produces a different distortion.
• Measure outside the process — at the load balancer and in the client — as the primary latency signal, with in-process metrics as a diagnostic.
• Instrument the accept path: connection establishment time and accept backlog depth, which exist before a request is served and are therefore visible during exactly the stalls that server-side latency cannot see.
• Prefer age of the oldest queued request to any percentile computed from completed work when alerting on saturation.

Industry example

The effect is why latency-measurement tooling was rebuilt around it: HdrHistogram and the load generators that adopted its correction (wrk2 among them) exist specifically to record from intended send time, and the practice of publishing latency at high percentiles with an explicit statement of the measurement model dates from the same period. Every benchmark published without saying whether it was open- or closed-loop is unfalsifiable, which is the practical reason the concept is worth knowing by name.

Failure scenarios

• A capacity sign-off based on a p99 that omitted every stall, followed by a launch that fails at a fraction of the tested rate.
• An SLO derived from load-test numbers that cannot be met by the same system under real arrivals.
• A garbage-collection or checkpoint pause that never appears in any test, because the generator paused with it.
• A regression that passes because the change made stalls longer and less frequent, which a closed-loop generator scores as an improvement.
• Two teams arguing with server-side and client-side numbers, neither of which is wrong.

Trade-offs

Open-loop generation is harder to run: the client must hold growing backlog, it needs more capacity than the system under test at overload, and the results are messier because an overloaded system produces genuinely unbounded latency rather than a tidy plateau. That messiness is the information. A closed-loop test is self-limiting by construction, which is why it always produces a comfortable number, and comfort is the thing being bought.

When not to use it

When the real client population is itself closed-loop, model it that way. A batch integration with 20 worker threads, an internal job runner, or an embedded device that waits for each response genuinely does stop sending when you stall, and simulating open-loop arrivals for it over-states the load. The correction matters when the arrival process is independent of your service's health — which is every user-facing system and most machine-to-machine traffic at scale.

Interview question

Q: Your load test reports a p99 of 30 ms at 5,000 rps. Production at the same rate has users reporting multi-second stalls, and the service's own latency metrics agree with the load test. Explain how all three can be true.

What a strong answer covers: the closed-loop generator stalling in lockstep with the server so the slow requests are never issued; the arithmetic of how few slow samples a stall actually contributes; why server-side metrics measure service time and not queueing before acceptance; the fixes — open-loop generation, recording from intended send time, measuring at the edge; and the alert that would have caught it, which is accept backlog depth or age of oldest queued request rather than any percentile of completed work.

Quick check

Quiz: Why does a closed-loop load generator understate tail latency? Because its virtual users stall alongside the server, so the requests that would have arrived during the stall are never sent and never measured.

Flashcard: Your load test and your server metrics agree, and users disagree. What is the likely cause? Coordinated omission plus service-time-only measurement: the generator stopped sending during stalls, and server-side latency starts at acceptance, so neither sees the queueing that users experienced.