Grey Failure
also called Partial Failure, Fail-Slow
A component that is degraded rather than down — slow, intermittently erroring, or failing for a subset of operations — which defeats health checks built for binary states.
The hardest failure mode at scale, and the one most monitoring is blind to. A disk that has become slow but still responds, a node with elevated packet loss, a dependency erroring on 5% of requests, a process that answers health checks while its work queue is stalled.
The defining property is differential observability: the component believes it is healthy and reports so, while its callers experience something quite different. Health checks pass, dashboards look normal, automated failover does not trigger, and the incident is diagnosed by a human noticing that a graph looks wrong.
Three responses that work better than trying to detect it precisely.
Measure from the caller's perspective. Client-observed latency and error rate per dependency reveal grey failure that server-side health checks never will.
Prefer generic mitigations over diagnosis. Draining a cell, evicting an instance, or shifting traffic away resolves a broad class of grey failure in minutes without knowing which one you have.
Make health checks reflect real work. A check that exercises the actual path — including its dependencies at the readiness level — catches more than one that confirms the process is running.