A request fans out to 100 services in parallel. Each responds within 10 ms for 99% of calls. What fraction of user requests are slow, and what do you do?

What the interviewer is testing

Whether you can do the tail amplification arithmetic and reach the right family of fixes.

The arithmetic

The probability that all 100 respond quickly is 0.99^100 ≈ 37%.

So roughly 63% of user requests hit at least one slow component. A one-in-a-hundred event at the component level is the common case at the user level.

This is why teams optimising averages see no improvement in user experience — the average was never the problem — and why p99 at the component level is the number that matters for a fan-out architecture, with p99.9 mattering for the services beneath it.

The fixes, in order of leverage

Reduce fan-out. The most effective and least used lever. 100 parallel dependencies for one request is itself the problem, and it usually indicates boundaries drawn too finely. Halving the fan-out does more than any latency optimisation.

Return partial results. Render with what arrived. This converts the tail from a latency problem into a completeness one, which is usually the better trade.

Hedged requests. Send a second copy to another replica after the p95 latency and take whichever responds first. Because only about 5% of requests hedge, the extra load is small and the effect on the tail is large.

Tighten per-dependency deadlines, so a slow component is abandoned rather than waited for.

Attack the sources of tail latency directly: garbage collection pauses, cold caches, noisy neighbours, and queueing at high utilisation.

The hedging caveat

Hedging requires idempotent operations and spare capacity. Under overload, latency rises across the board, a large fraction of requests exceed the threshold, and hedging amplifies the load exactly when there is none to spare. It needs a hedge budget — a hard cap of around 5% of traffic, enforced globally — or it inverts from an optimisation into a collapse.

What a strong answer adds

The insight that makes hedging work: tail latency is usually caused by a slow server, not a slow request — a node in garbage collection, sharing a host, or with a cold cache. The same request would have been fast almost anywhere else, which is precisely what hedging exploits.

Common weak answers

Optimising the average response time of each service. Adding retries, which do not help when the dependency is slow rather than failing.