Gray Failure

Systems are built to handle crashes, and crashes are the easy case: the node stops, the health check fails, traffic moves. Gray failure is the hard case, and it causes a disproportionate share of long incidents.

The defining property is differential observability — the component's own view of its health disagrees with its users'. A node reports healthy because its health endpoint returns 200 while its disk is failing, its network path is dropping 5% of packets, one dependency is timing out, or a memory leak has pushed it into constant garbage collection. Load balancers keep sending traffic to it, retries land on it again, and the aggregate error rate stays below alerting thresholds while a subset of users has a broken experience.

The detections that work: health checks that exercise the real dependency path rather than returning a constant, comparison of per-instance metrics against the fleet so an outlier is visible even when the aggregate is fine, client-side success rates as the authoritative signal rather than server-side, and latency percentiles per instance rather than pooled.

The response that works: make removal easy and cheap. Outlier detection that ejects an instance whose error rate diverges from its peers, and a bias towards restarting or replacing rather than diagnosing in place, because a fleet member that is behaving strangely costs more to investigate live than to replace.