A search tier fans each query out to 60 shards and waits for all of them. Shards are spread over three availability zones. One zone's instances become 30 ms slower than the others - not failing - after a kernel upgrade. What happens to query latency?

Second by second

Nothing fails, and every query gets slower, which is why this class of fault is harder than an outage. A scatter-gather query that waits for all 60 responses finishes when its slowest shard finishes. One third of the shards live in the degraded zone, so the probability that a given query touches at least one slow shard is effectively 1: with 20 of 60 shards affected, essentially every query inherits the 30 ms.

The arithmetic generalises and is worth memorising. If a single shard exceeds some latency with probability p, a 60-way fan-out exceeds it with probability 1 − (1 − p)^60. At p = 1% that is 45%. A one-in-a-hundred event at the shard becomes a one-in-two event at the query. This is tail amplification, and it is the reason a fan-out architecture is so sensitive to small regressions.

Where it amplifies

The 30 ms is the start. Query latency rises, so concurrency rises for the same arrival rate by Little's Law: at 2,000 queries per second, adding 30 ms adds 60 in-flight queries, each holding a thread, a buffer and 60 open shard requests. If the fan-out layer has a bounded request pool, it saturates, and now queries wait before being sent as well as after. Callers with a 250 ms deadline begin timing out and retrying, which adds offered load on top of the regression. The system amplifies a 30 ms degradation into a latency collapse through queueing, not through the fault itself.

What the user sees

For the first minutes: a slower search box. Then, as retries stack, intermittent errors across all zones, including the healthy ones, because the saturated resource (the fan-out layer's pool) is shared. The dashboards show the degraded zone's shard latency up 30 ms and every zone's query latency up far more, which is the signature that distinguishes amplification from a local fault.

What stops it

• Partial results with a deadline. Return after waiting a fixed budget with however many shards answered, and mark the response as incomplete. A 95%-complete result in 80 ms beats a complete one in 400 ms for almost every search product, and this single change breaks the dependence on the slowest shard.
• Hedged requests to a replica of the slow shard, issued only after the p95 of normal shard latency has elapsed, so the extra load is a few per cent. Cancel the loser.
• Passive outlier ejection at the fan-out client: track per-instance success and latency, and temporarily stop sending to instances whose latency is an outlier against their peers. This is what finds a kernel regression nobody announced.
• Zone-aware routing so each fan-out layer prefers shards in its own zone. It does not fix a slow zone, but it confines the damage to the queries served from that zone rather than smearing it across all of them.

What would have to be true for it to self-heal

Only that the fan-out layer rejects work faster than it accumulates it: bounded in-flight requests, a deadline that is enforced rather than hoped for, and a retry budget expressed as a fraction of successful traffic rather than a per-call count. Without those three, no amount of healthy capacity in the other two zones prevents the collapse.

When not to reach for hedging

Hedging and partial results both cost something, and the standard reference for them is explicit about it: Dean and Barroso's tail-at-scale paper (CACM, 2013) describes hedged requests as worth the extra load precisely because the duplicate is issued only after the normal case has already failed to complete. Choose hedging when the operation is idempotent and cheap to duplicate; it flips when the request mutates state or costs real money, as a payment authorisation or a model inference does, where the right answer is a tighter deadline and a smaller fan-out instead. Partial results are the wrong answer when the query is a financial aggregate rather than a search ranking: a 95%-complete balance is not a degraded answer, it is a false one.