One instance in a fleet of fifty is returning correct responses very slowly. Health checks pass and it stays in rotation. How do you detect and handle this?

Why it is not detected

Differential observability. The instance believes it is healthy and reports so; its callers experience something else. A health check that confirms the process is running and can answer a trivial endpoint will pass indefinitely on an instance whose disk is failing slowly, whose garbage collector is thrashing, or whose connection pool is exhausted.

Aggregate metrics hide it too: one instance in fifty degrading moves the fleet's p99 by an amount that looks like noise, while 2% of users have a bad experience every time they land there.

Detection

Per-instance metrics, compared against the fleet. The signal is not an absolute threshold — it is an outlier. Latency, error rate and saturation per instance, alerted when one deviates significantly from the median of its peers. This is the single most effective change, and most monitoring setups do not do it because they aggregate before they alert.

Client-side observation. Callers know which instance served them and how it went. Load balancers and service meshes expose per-upstream latency and error rates, and that view is more honest than the instance's self-report.

Health checks that exercise real work — a readiness check that touches the dependency path and takes a realistic action, rather than returning 200 from a trivial handler.

Handling

Outlier ejection, which is the correct automatic response and is built into most meshes and load balancers: an instance that deviates from its peers is removed from rotation for a period, then tentatively returned. It requires no diagnosis and resolves a broad class of grey failure in seconds.

Two guards it needs: a cap on how many instances may be ejected simultaneously (typically a small percentage, so a fleet-wide problem does not eject everything), and gradual return so a still-sick instance is not immediately loaded.

Then let it be replaced. If ejection recurs for the same instance, terminate it and let the platform provide a new one. Diagnosing a single sick instance is rarely worth the time when replacing it is cheap.

The wider principle

Prefer generic mitigations that need no diagnosis. Eject the instance, drain the cell, roll back the deploy, shift the traffic. Each resolves many possible causes in minutes, whereas identifying which grey failure you have takes hours. Diagnose afterwards, from the evidence, with the site healthy.

What a strong answer adds

Noting that this is why liveness and readiness must be separate: an instance that is slow should be removed from rotation (readiness) without being restarted (liveness) until you know more — and a liveness check that tests dependencies will restart the entire fleet during a downstream incident, turning a degradation into an outage.