You add hedged requests to cut tail latency. It works well, then during a traffic peak the service collapses. Explain.

What the interviewer is testing

Whether you know the condition under which hedging inverts from a latency optimisation into an amplifier.

The mechanism

Hedging sends a second copy of a request when the first has not responded by a threshold — typically the p95 latency. Under normal conditions only about 5% of requests hedge, so extra load is about 5% and the effect on the tail is large.

Under overload, latency rises across the board. Suddenly a large fraction of requests exceed the threshold, so a large fraction hedge. Load increases substantially, which increases latency further, which causes more hedging.

The loop is self-reinforcing and it fires precisely when there is no spare capacity.

The fixes

A hedge budget. Cap hedged requests as a percentage of total traffic — typically 5% — enforced globally rather than per request. When the budget is exhausted, no further hedging occurs regardless of latency. This single control prevents the collapse and is the most important one.

Adaptive threshold computed from live traffic, so the trigger rises with genuine load rather than staying at a value that everything now exceeds.

Disable hedging under load shedding, tied to the same signals that trigger shedding.

Tied requests, where the two replicas are told about each other so whichever starts first cancels the other, cutting the wasted work substantially.

The conditions that must hold anyway

Idempotent operations, since both copies may execute. Spare capacity as a standing assumption — which is exactly what fails under peak. And a threshold derived from live percentiles rather than a fixed value that becomes wrong as conditions change.

What a strong answer adds

The underlying insight: tail latency is usually caused by a slow server rather than a slow request — a node in garbage collection, sharing a host with a noisy neighbour, or with a cold cache. Hedging exploits that, which is why it works so well and why it stops working when the slowness is systemic rather than per-instance. Distinguishing the two is what tells you whether hedging is the right tool at all.

Common weak answers

Removing hedging entirely, which loses a genuine improvement. Raising the threshold without a budget.