A search platform's index-serving region becomes degraded but not fully down - elevated latency and partial errors. Should traffic fail over automatically? Analyse the risks either way.

Why partial degradation is the hard case

Automatic failover is straightforward for a clean failure: the region is unreachable, move traffic. Partial degradation — the gray failure — is much harder, because the region is still serving most requests correctly and the surviving regions may not have capacity for its full load.

Risks of failing over automatically

Capacity. The remaining regions must absorb the load. If they were sized for their own traffic plus a margin, absorbing a full region may overload them — turning a partial degradation affecting some users into a total outage affecting all.

Cold caches and cold indexes. For a search platform this is severe: the receiving region's caches are warmed for its own query distribution. Absorbing another region's traffic means a large miss rate, higher backend load, and higher latency exactly when the system is under stress.

Flapping. If the degradation is intermittent, automated failover moves traffic back and forth, and each move costs cache warmth and connection re-establishment. Oscillation is often worse than either stable state.

Correlated cause. If the degradation is caused by a bad deploy or a bad configuration, the other regions have it too, and failing over spreads the load onto systems about to exhibit the same problem.

Risks of not failing over

Sustained user-visible degradation for everyone routed to that region, potentially for hours if human judgement is required and nobody is confident.

Gray failures are under-detected. They often do not trip alarms designed for outages, so "wait for a human" frequently means "wait until customers complain".

The design that resolves it

Partial, gradual, automatic shedding rather than binary failover.

• Move a fraction of traffic, measure, move more. This bounds the capacity risk and lets the receiving region warm gradually.
• Trigger on user-visible signals — latency percentiles and error rates measured at the edge — not on host health, because gray failures pass host health checks.
• Hysteresis, so returning traffic requires sustained recovery rather than a momentary improvement.
• A capacity check as a precondition. Automation must know whether the destination can absorb the traffic and refuse or partially shed if not. Automation that can create a worse outcome than the problem must be able to decline.
• Shed by request class. For search, the expensive tail — deep pagination, complex queries, rarely-used verticals — can be shed while common queries continue. Losing a feature beats losing the product.
• A human-visible kill switch, so an operator can stop or force the automation without a deploy.

The general principle

Automate the response you would want at 3 a.m. with incomplete information, and bound its blast radius. Full automatic failover is right when the destination is provably capable and the failure is clean. For gray failures with capacity constraints, graduated shedding with an operator override is safer than either extreme — and the failure mode to fear most is automation confidently making things worse.