Latency

Definition

Latency is the elapsed time of an operation. The critical property is that it is a distribution, and describing it with a single number destroys the information that matters.

Why the tail dominates

An average latency of 200 ms is compatible with 90% of requests at 50 ms and 10% at 1.5 s. The average describes nobody's experience.

Worse, in a system with fan-out, the tail becomes the norm. If a page requires 20 parallel backend calls and each has a 1% chance of exceeding 1 second, the probability that at least one does is about 18%. So a p99 at the service level becomes a p82 at the page level. This is the mechanism by which tail latency at one layer becomes typical latency at the next, and it is why fan-out architectures must care about p99.9 rather than p95.

Where latency comes from

• Network round trips, bounded by physics — roughly 150 ms intercontinentally, unimprovable.
• Queueing. As utilisation rises, queueing delay rises non-linearly. At 80% utilisation waiting time is roughly four times the service time; at 90% it is nine times. This is why running systems "efficiently" at high utilisation produces terrible latency, and why headroom is a latency feature.
• Serialisation — encoding, decoding, compression.
• Contention — locks, connection pool waits, garbage collection pauses.
• Cold caches, cold starts, JIT warm-up, which land squarely at p99.

Measuring it properly

• Measure where the user is. Server-side latency excludes DNS, TLS, the network and rendering — most of what the user experiences.
• Percentiles, and know they do not average. The mean of ten instances' p99 values is not the fleet p99; aggregate the histograms.
• Report the condition. "p99 under 300 ms" is meaningless without "at what load, measured where".
• Look at the distribution shape. Bimodal latency — a fast path and a slow path — is very common (cache hit versus miss) and is invisible in any single percentile.

Failure scenarios

• Averages reported to stakeholders, so a real problem is invisible.
• p99 measured per instance and averaged, producing a number that is not a percentile.
• Latency budget never stated, so a new dependency consuming a third of it goes unnoticed.
• High utilisation pursued as efficiency, producing queueing delay that no code optimisation fixes.

Interview question

"A page makes 20 parallel backend calls, each with a p99 of 1 second. What is the page's latency profile and what would you do about it?"