Outlier Detection

A health check asks a host whether it is well. Outlier detection asks whether a host is behaving differently from the others - which catches the faults a health endpoint cannot see: a host failing 30% of requests, one whose disk is slow, one with a stale configuration, one on a degraded network path. The host answers its own check correctly and serves users badly.

The signal is real traffic, which is both the strength and the catch: the first clients to discover a bad host are users. Passive detection is a damage-limiting mechanism, not a prevention mechanism, which is why it belongs alongside active health checks rather than instead of them.

Why it matters

The faults it catches are the common ones. Hard-down hosts are handled by every load balancer ever built; grey failure - partial, intermittent, slow - is what actually produces incidents, and it is defined relative to peers rather than against an absolute threshold. A fleet where every host fails 30% of requests is a systemic problem that ejection must not try to fix; a fleet where one host does is exactly what ejection is for.

That distinction is why the configuration details matter more than the feature.

Implementation patterns

• Use a success-rate trigger, not only consecutive errors. Envoy's success-rate detection compares each host against the fleet's distribution and ejects statistical outliers; consecutive-5xx never fires for a host failing 30% of requests at moderate traffic.
• Add a latency trigger or a tight timeout, because a slow host is the most common outlier and produces no errors at all.
• Cap maximum ejection at around 10%. Half the fleet is not an outlier population; it is an incident, and ejecting into it concentrates load on the survivors.
• Keep the panic threshold enabled. When the healthy share falls below it (50% by default in Envoy), traffic is distributed to all hosts including unhealthy ones. This looks wrong and is right: when most hosts appear unhealthy, the health signal is the most likely thing that is broken, and overwhelming the few that still pass is strictly worse than spreading the load.
• Bound retries with a budget - retries as a percentage of active requests, 20% being a common starting point - because unbudgeted retries triple load during the fault and accelerate ejection, shrinking the fleet while load rises.
• Eject for a growing interval (base interval multiplied by consecutive ejections) and return hosts gradually, so a flapping host is not re-admitted into full traffic.
• Alert on ejection rate and on panic mode. Both are statements about the fleet, not about a host.

Industry example

Envoy, open-sourced by Lyft in 2016 to move networking concerns out of services, is where most teams meet this: its outlier-detection module ships consecutive-5xx, success-rate and latency triggers, ejection ceilings, growing ejection intervals and the panic threshold, and its defaults are deliberately conservative because the correct settings are a property of the fleet rather than of the proxy. The same primitives appear in gRPC's client-side load balancing and in cloud load balancers as "connection draining plus automatic deregistration", with less configurability and the same underlying choices.

Failure scenarios

• A bad deployment ejects the fleet. Half the hosts fail 40% of requests, retries triple the load, ejection removes up to the ceiling, survivors saturate, and the system enters panic mode having first ensured the remaining hosts were overwhelmed.
• Fast-failing hosts attract traffic under least-request balancing while ejection has not yet fired, because their in-flight count is lowest.
• A canary or shard-holding host is ejected for being different, which it was designed to be.
• Small fleets destabilised: on 4 hosts, one ejection is a 25% capacity loss and normal variance looks like an outlier.
• Flapping, where a host is ejected, returned at full weight, fails again, and the cycle adds latency on every round.

Trade-offs

When not to use it

Below roughly five hosts, leave it off: one ejection is a fifth of capacity, and variance is indistinguishable from an outlier. Turn it off for deliberately heterogeneous backends - canaries, shadow versions, hosts serving different shards - because it will find the difference you created. And do not treat it as a substitute for a rollback: when the fault is in the deployment rather than in a host, the fleet-wide fix is to roll back, and an aggressive ejection policy will have made the incident worse while the rollback was being considered.

Interview question

Q: A mesh ejects a host after five consecutive 5xx responses, with a 50% maximum ejection, three retries on any 5xx and no active health checks. Walk me through what happens when a deployment makes 40% of requests fail, and what you would change.

What a strong answer covers: the amplification sequence - retries triple load, uneven consecutive-5xx firing, up to half the fleet ejected, survivors saturated, panic mode entered after the damage; replacing retry counts with a budget and restricting retries to idempotent and connect-time failures; a 10% ejection ceiling; adding success-rate and latency triggers plus active health checks; keeping the panic threshold and alerting on it; and the framing that mesh resilience settings are a rollout concern reviewed by imagining a bad deploy rather than a dead host.

Quick check

Quiz: Why keep sending traffic to unhealthy hosts once most of the fleet is marked unhealthy? Because the health signal is then the most likely thing broken, and concentrating all load on the few hosts still passing would overwhelm them immediately.

Flashcard: What does consecutive-5xx ejection miss? A host failing a minority of requests and a host that is slow rather than erroring - the two most common grey failures, which need success-rate and latency triggers.