Review this service mesh configuration. Passive outlier detection ejects a host after 5 consecutive 5xx responses with a maximum ejection of 50% of hosts; there are no active health checks; retries are set to 3 attempts on any 5xx with no budget; and the panic threshold is left at its default. What would you change?

What is actually required

Two separate jobs are being conflated. Outlier detection removes a host that is behaving differently from its peers; health checking removes a host that says it is unwell. The configuration has one and not the other, and it has a retry policy that will interact with both during a bad deployment.

What I would remove

The unbudgeted retries. Three attempts on any 5xx multiplies offered load by up to 3x at exactly the moment a dependency is failing, and when combined with ejection it accelerates the loss of capacity: each retry that fails pushes its host closer to ejection, so the fleet shrinks while load triples. Replace the count with a retry budget - retries capped as a percentage of active requests, 20% being a common starting point - and restrict retries to idempotent methods and to connect-time failures, where they are genuinely safe.

The 50% maximum ejection goes too. Half the fleet is not an outlier population; it is an incident. A ceiling in the 10% range keeps ejection as a mechanism for picking off genuinely odd hosts rather than a mechanism for amplifying a systemic fault.

The one change that matters

Add a success-rate-based trigger alongside the consecutive-5xx one, and add active health checking. Consecutive-5xx catches a host that is hard-failing and misses the two most common real faults: a host that fails 30% of requests (never five in a row at moderate traffic) and a host that is slow rather than erroring. Envoy's success-rate detection compares each host against the fleet's distribution and ejects statistical outliers, which is the correct shape for "different from its peers". Slow-host detection needs a latency-based trigger or a timeout tight enough to turn slow into an error.

Active health checks matter for a different reason: passive detection only learns about a host from real user requests, so the first clients to meet a broken host are customers. Active checks find it before traffic does, and they are the only mechanism that can return a host to service safely.

What I would leave, even though it looks odd

The panic threshold at its default. Envoy's panic mode sends traffic to all hosts, including unhealthy ones, when the healthy share falls below a threshold (50% by default), and engineers routinely want to disable it because distributing traffic to known-bad hosts is obviously wrong. It is not. When most of the fleet appears unhealthy, the most likely explanation is that the health signal is broken, not the fleet - a bad config push, an expired certificate, a dependency outage making every host fail its check. In that state, spreading traffic across everything is strictly better than concentrating it on the three hosts that still pass, which would be instantly overwhelmed. Keep it, and make sure it is alerted on.

How I would argue this in the review

By walking through one scenario rather than debating settings, and naming the costs: active health checks add a request per host per interval and a failure mode of their own when the check is deeper than the service, while a 10% ejection ceiling means a genuinely bad fifth of the fleet keeps taking traffic until the deployment rolls back. Those are the prices, and they are lower than the alternative: a deployment ships a bug that makes 40% of requests fail. With the current configuration, retries triple the load, consecutive-5xx ejection fires unevenly, up to half the hosts are removed, the survivors saturate, and the mesh enters panic mode anyway - having first ensured that the remaining hosts were overwhelmed. The proposed configuration sheds the retry amplification, ejects at most a tenth, and relies on the deployment's own rollback. Mesh resilience settings are a rollout concern, not merely a reliability concern, and the review that catches this is the one that imagines a bad deploy rather than a dead host.

When not to eject at all

Ejection assumes the fleet is homogeneous and the fault is local. Prefer no outlier detection at all on a fleet of fewer than about 5 hosts, where removing one is a 20% capacity loss and a statistical outlier cannot be distinguished from normal variance. Turn it off too for a service whose backends are deliberately heterogeneous - a canary, a shadow version, or hosts holding different shards - because ejection will find the difference it was designed to find and remove the thing you were testing. Envoy's implementation, open-sourced by Lyft in 2016 and now the common substrate for this behaviour, is configurable precisely because the right settings are a property of the fleet rather than of the proxy.