One instance in a fleet of 20 is failing 30% of its requests. Health checks pass, aggregate error rate is 1.5%, no alerts fire. How do you detect and handle this?

What the interviewer is testing

Whether you know gray failure and its defining property — differential observability.

Why nothing fired

Aggregate metrics hide it. One instance failing 30% of its share contributes 1.5% to the fleet total, below any sensible alerting threshold. Meanwhile 1 in 20 users is having a substantially broken experience, and retries land on the bad instance again.

Health checks pass because they usually test something trivial — that the process responds — rather than the path that is actually failing. The instance's own view of its health disagrees with its users'.

Typical causes: a failing disk, a degraded network path dropping packets, one dependency connection pool exhausted on that host, a memory leak causing constant garbage collection, or a stale cached credential.

Detection

Per-instance metrics compared against the fleet. The signal is not the absolute error rate, it is that one instance diverges from its peers. Outlier detection on per-instance error rate and latency finds this immediately and finds it for causes nobody anticipated.

Health checks that exercise the real dependency path rather than returning a constant — a shallow check for liveness, a deeper one for readiness.

Client-side success rates as the authoritative signal, since the caller's view is the one that matters and it disagrees with the server's.

Handling

Outlier ejection at the load balancer or mesh: automatically remove an instance whose error rate diverges significantly from its peers, and re-admit it cautiously.

Bias towards replacement over diagnosis. A fleet member behaving strangely costs more to investigate live than to replace. Terminate it, let the fleet replace it, and analyse the captured telemetry afterwards.

What a strong answer adds

Applying the same reasoning one level up: gray failure occurs across availability zones and regions too, where one zone is degraded rather than down, and health-check-based failover expecting a binary state does not trigger. Per-zone metric comparison is the equivalent detection.

Common weak answers

Lowering the alert threshold, which produces noise across the fleet. Relying on health checks that did not detect it.