One instance passes its health checks but serves requests ten times slower than its peers. Why is it still receiving traffic and what would you change?

What is being tested

Understanding of grey failure — degraded but not dead — which binary health checks are structurally incapable of detecting.

Why it still receives traffic

A health check answers a binary question: is this instance alive? A slow instance answers "yes", because it is alive. It just takes 2 seconds instead of 200 ms.

Worse, with round robin balancing it receives exactly the same share of requests as its healthy peers, because round robin distributes requests, which is the wrong unit — the right unit is work or in-flight time. The slow instance accumulates a queue while its peers idle.

Grey failure is far more common and far more damaging than hard failure, because hard failure is detected and handled automatically, while grey failure degrades user experience indefinitely and is usually noticed by customers rather than by monitoring.

What to change

1. Latency-aware balancing. Least response time, or least outstanding requests. Both naturally route away from a slow instance because it accumulates in-flight requests. This is a configuration change on most balancers and it is the highest-value fix.

2. Outlier detection / passive health checking. Track per-instance error rate and latency against the fleet, and eject an instance that deviates significantly, returning it gradually after a cool-off. This is the mechanism specifically designed for grey failure.

3. Deep health checks — carefully. A health check that verifies the instance can actually do useful work catches more than a liveness ping. But a health check that tests dependencies creates a much worse failure: a downstream slowdown makes every instance report unhealthy, the platform removes them all, capacity collapses, and a partial problem becomes a total outage. Liveness tests the process; readiness may test dependencies, and they must be separate endpoints.

4. Client-side or request-level balancing. Netflix's client-side approach exists partly for this reason: the caller knows which instances have recently been slow for its own requests, which no central health check can know. It can also prefer instances in its own availability zone, avoiding a cross-zone hop and its charges.

The cost is that the logic must exist in every client language and cannot be upgraded without redeploying every caller — which is precisely the pressure that later drove the industry toward sidecar proxies, where the same logic sits outside the application.

Then find the cause

Ejecting the instance stops the bleeding; it does not explain it. Usual suspects: a noisy neighbour on shared hardware, a degraded disk, an instance that has been running long enough to accumulate a memory or connection leak, a JIT or cache warm-up state that never completed, or a connection pool exhausted against one downstream.

Automated ejection plus an alert is the right combination. Silent ejection hides a recurring problem; alerting without ejection leaves users suffering while someone investigates.