A video platform's request path fans out to twenty backend services in parallel. Each has a p99 of 50 ms. Why is the overall p99 far worse than 50 ms, and what fixes it?
Show the full answer Hide the answer
Why the tail amplifies
A parallel fan-out completes when its slowest component completes. If each of twenty calls independently has a 1% chance of exceeding 50 ms, the chance that at least one does is 1 − 0.99²⁰ ≈ 18%.
So a p99 of 50 ms per component produces roughly an 18th-percentile chance of exceeding 50 ms overall. Stated the other way: the overall p99 is determined by each component's p99.95, not its p99.
This is why tail latency dominates at scale. In a system with wide fan-out, ordinary requests routinely encounter at least one slow component, and improving the median does nothing.
What causes the individual tails
Rarely capacity. Usually per-node interference:
- Garbage collection pauses.
- Compaction or background maintenance.
- Cache misses on a cold or evicted key.
- Queueing behind an expensive request.
- A single degraded instance in a large fleet.
- Network retransmission on one path.
These are transient and node-specific, which is precisely what makes them addressable by redundancy rather than by tuning.
What fixes it
1. Hedged requests. Send the request; if no response arrives within, say, the p95, send a duplicate to another replica and take the first answer. This converts a component's p99 into approximately its p95 squared, which is a dramatic improvement. Cost: a small percentage of extra load, tunable by the hedge delay.
2. Tied requests. Send to two replicas with a cancellation between them, so the loser stops work. More efficient than hedging, requiring replica cooperation.
3. Reduce fan-out width. Twenty parallel calls is itself the problem. Combining calls, caching aggregates, or precomputing the combined result removes terms from the amplification.
4. Do not require all responses. If the page is useful with eighteen of twenty results, apply a deadline and return partial results. This bounds latency by construction rather than by hoping every component is fast.
5. Reduce per-node variance. Runtime tuning to shorten pauses, separating background work from serving, and removing single degraded instances quickly. Addressing the cause is better than compensating for it.
6. Micro-partition and rebalance, so a slow node holds a small share of any request's work rather than a large one.
The measurement discipline
Measure the end-to-end percentile, not the component percentiles. A dashboard showing twenty services each at a healthy p99 will look entirely fine while the user-facing p99 is several times worse — and the discrepancy is arithmetic, not a monitoring gap.