Every distributed system runs a machine whose job is to declare other machines dead and act on the verdict without asking anyone. This guide reconstructs how that machine is actually built, from six published incidents, five production codebases, four papers and two decades of design records, and lands on the rule the survivors converged on: the detector gets a budget for being wrong, and the budget is enforced on the action, not the signal.
The problem, stated without naming a product: from incomplete signals, decide whether a peer that stopped answering is dead or merely slow, then act on that verdict automatically, without the action making the situation worse.
On 20 October 2025, hours after the DynamoDB DNS record that triggered the incident had been repaired, AWS was still losing capacity in us-east-1. The reason, according to AWS's own postmortem, was the Network Load Balancer's health-check subsystem: checks "alternated between failing and healthy, which caused NLB nodes and backend targets to be removed from DNS, only to be returned to service when the next health check succeeded." The servers were fine. The thing deciding whether they were fine was not, and it was wired directly to the lever that removes capacity. That wiring is the subject of this guide.
A failure detector can be wrong in two directions, and the costs are wildly asymmetric. A false negative leaves a dead server in rotation; David Yanacek's account in the Amazon Builders' Library (2019) explains why this is worse than it sounds: a server that fails requests fast attracts more traffic under least-requests balancing, "creating a 'black hole' in the service fleet." A false positive removes a healthy server, which is cheap, once. The disaster case is the correlated false positive: one shared cause (a dependency blip, a saturated network, a bug in the checker itself) makes every server fail its check in the same minute, and an unbudgeted actor removes them all. Five of the six incidents in this guide are that shape.
Between 2012 and 2025, at least five independent teams converged on the same guardrail under five different names: Netflix's Eureka stops expiring registrations when renewals drop below 85% ("self-preservation", in the code since 2012); Envoy caps passive ejection at 10% of a cluster and ignores health data entirely below 50% availability ("panic threshold"); HAProxy ships the same idea, and Slack credits it by name ("panic mode") for turning their January 2021 outage from down into degraded; Route 53, ALB and NLB "fail open" when zero targets are healthy; and AWS's October 2025 remediation adds "velocity control" to limit how fast health checks can remove NLB capacity. Five names, one rule: cap what the failure detector may do before you trust what it says. The most widely deployed health checker on earth, the kubelet, has no such cap; the 2018 issue asking for one (kubernetes/kubernetes#66230) is still open, frozen.
Scope: this guide covers failure detection between servers inside a system, meaning health checks, probes, gossip-based membership and the automated actions wired to them. It deliberately does not cover human-facing monitoring and alerting, consensus-protocol correctness, or what the clients of a failing service should do; that last problem (retries, backoff, metastable overload) has its own dig in this collection. Where the record runs out, the page says so: the most conspicuous gap is named in section 04.
The common shape across AWS, Google, Envoy, Kubernetes, Consul, Cassandra and Eureka: signals feed a suspicion score, the score crosses a threshold, and then, in every mature design except one, the resulting action passes through a gate that limits how much the detector may do at once.
Observe. Four signal families recur. Active probes are a synthetic request on a timer: the NLB and HAProxy model, and the kubelet's liveness, readiness and startup probes. Passive detection watches real traffic instead: Envoy's outlier detection ejects a host on consecutive 5xx or on statistical deviation from the cluster's success rate, which means it costs nothing extra and sees what users see, but also means it is blind on idle clusters. Self-reporting inverts the direction: Google's RPC servers enter a "lame duck" state in which, per the SRE book's load-balancing chapter, the task "is listening on its port and can serve, but is explicitly asking clients to stop sending requests"; gRPC standardised the same move as the health-checking protocol's NOT_SERVING status. And peer gossip makes every node a checker of a few random others: SWIM and its descendants in Consul's memberlist and Cassandra's gossiper, adopted because, as the SWIM paper (2002) argues, central all-to-all heartbeating imposes network load that grows quadratically with group size.
Judge. The naive judgment is a counter: Kubernetes convicts after 3 consecutive probe failures, 10 seconds apart, 1-second timeout, all defaults readable in types.go. The refined judgment is a continuous suspicion score: Cassandra's FailureDetector.java implements Hayashibara's 2004 phi accrual detector, which models heartbeat inter-arrival times as a distribution and emits a suspicion value that rises smoothly as silence grows, convicting at a configurable phi (default 8 in cassandra.yaml). The difference matters on jittery networks: a fixed timeout encodes one assumption about latency; phi re-learns the assumption continuously. The 2017 Lifeguard paper from HashiCorp adds the judgment nobody else had written down: the checker should also suspect itself. Serf and Consul's memberlist scores its own responsiveness (an "awareness" counter in the code) and lengthens its probe timeouts when it is the degraded party; the paper measures a reduction of false positives "by over 98%" at the median-latency-neutral setting. Slow message processing at the observer, not death at the observed, was most of the noise.
Gate, then act. This is the layer this guide exists to point at, and the one I have to name myself because every source names it differently: call it the eviction budget. Envoy enforces it twice, as a per-sweep cap (at most 10% of a cluster ejected by outlier detection, per the proto default) and as a global floor (below 50% available hosts the load balancer "will disregard health status", with the docs stating the purpose plainly: "to avoid a situation in which host failures cascade throughout the cluster as load increases"). Eureka's registry stops expiring instances entirely when fewer than 85% of expected heartbeat renewals arrive, because, per the code's own javadoc, "eureka perceives this as a danger", the danger being a network problem masquerading as mass death. The AWS load balancers fail open at zero healthy targets. Each of these is the same statement in different syntax: past a threshold of agreement, the dead verdicts stop being information about servers and start being information about the detector, so the action lever locks.
Central checker fleet: NLB, Route 53. Every client checks for itself: Envoy, HAProxy, gRPC client-side health (design record A17, 2018). The node's own agent: kubelet. Random peers: SWIM, Consul, Cassandra. Per-client checking gives each client a private, partition-tolerant view; central checking gives one consistent view and one correlated failure mode, which is what flapped in the AWS October 2025 incident.
Stop routing (all load balancers), restart the process (kubelet liveness only), or fail over state (GitHub's Orchestrator, Consul leader election). The severity ladder matters because only the first action is cheaply reversible. GitHub's 2018 postmortem is the canonical case of a correct verdict wired to a hard-to-reverse action.
Microsoft's HotOS 2017 paper names the failure class every binary check misses: "differential observability", where "the system's failure detectors may not notice problems even when applications are afflicted by them". Roblox's 73-hour outage is this exactly: leaders that were alive, checkable and useless. Azure's operational answer (SREcon24 talk below) is to compare the app's view with the checker's view instead of trusting either.
Four forks, each with the condition that flips it. The pattern across all four: the more powerful the action, the dumber the check should be, and the tighter the budget.
| Decision | Chosen by | Rejected | Because | Evidence |
|---|---|---|---|---|
| Cap the detector's removal rate | Envoy (10%/sweep), Eureka (85% floor), AWS NLB (velocity control, being added) | Unconditional per-verdict action | Correlated verdicts indict the checker, not the fleet | AWS, 2025 |
| Probe depth for kill actions | Kubernetes docs, Zalando, Breck: process-local only | Deep liveness | A dependency blip restarts the whole fleet at once | k8s docs |
| Slow starters | startupProbe (KEP-950, v1.16+) | initialDelaySeconds; huge failureThreshold | Both trade away deadlock detection; the KEP says so explicitly | KEP-950 |
| Planned exits | Self-report: lame duck, gRPC NOT_SERVING | Letting the checker discover the drain | Propagates in "1 or 2 RTT" instead of a probe period, zero errors | SRE book |
| Membership at scale | Gossip with suspicion (SWIM lineage) | Central all-to-all heartbeats | Quadratic network load, or worse detection latency | SWIM, 2002 |
| Client-side health for gRPC | Reuse server health service per-connection | Look-aside balancer as sole checker | Not every deployment has a balancer in the path | gRPC A17, 2018 |
Six published incidents, in three classes: the detector condemned the fleet; the detector believed the fleet; the detector was right and the action was still the outage.
Slack's January 4, 2021 postmortem is the same setup as Class 1 with the opposite ending. AWS network saturation made health checks fail en masse against healthy web instances, exactly the correlated false positive. But Slack's HAProxy tier had "a feature called 'panic mode' which balances requests across all instances when many are failing health checks". The detector was just as wrong as AWS's NLB checker in 2025; the budget refused to act on it, and Slack spent the morning degraded instead of dark. Same failure, one design difference, different day.
One absence is worth stating plainly. Among these six accounts, and in the wider corpus behind this page, no operator postmortem attributes an incident to the failure mode health checks nominally exist for: a genuinely dead server left in rotation because nothing noticed. Yanacek describes the class from inside Amazon (his opening example is a single server whose disk froze overnight and which "black-holed" its share of traffic), but as a war story motivating checks, not a published incident. Read that two ways at once: basic detection is a solved problem, and the residual risk has moved entirely into the reaction. The published record says you are far more likely to be taken down by your health checks than saved from a downed server by them, because the saves are silent and the misfires make the news.
Defaults you will inherit, budgets others chose, and what wrong verdicts have cost. Measured and reported figures only; anything derived says so.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Default panic threshold | 50% | Envoy | Below this availability, health data is disregarded | 2026 | docs |
| Default max ejection | 10% | Envoy | Cap on cluster share removable by outlier detection | 2026 | proto |
| Self-preservation floor | 85% | Netflix Eureka | Renewal rate below which expiries stop entirely | 2026 | DefaultEurekaServerConfig |
| Default probe verdict | 3 × 10 s, 1 s timeout | Kubernetes | Consecutive failures, period, and the timeout that bites under CPU throttling | 2026 | types.go |
| Conviction threshold | phi = 8 | Cassandra | Continuous suspicion score; "most users should never need to adjust this" | 2026 | cassandra.yaml |
| False-positive reduction from self-suspicion | >98% (50×) | Lifeguard eval | At β=6, median detection latency unchanged | 2017 | paper |
| Lame-duck propagation | 1–2 RTT | Self-reported drain reaches all clients, vs a probe period for discovery | 2016 | SRE book | |
| Partition that triggered failover | 43 s | GitHub | Automated promotion; degraded service for 24 h 11 m | 2018 | postmortem |
| Gray-failure outage | 73 h | Roblox | Every node alive; leadership flapping among slow leaders | 2021 | postmortem |
| Regional event window | ~14.5 h | AWS us-east-1 | 11:48 p.m. Oct 19 to 2:20 p.m. Oct 20 PDT; NLB check flapping inside it | 2025 | postmortem |
| Fail-closed checker outage | ~7.5 h | Google Cloud | Crash-looping Service Control; ~40 min to roll the red-button disable | 2025 | incident report |
Two derived numbers worth doing on your own fleet. First, worst-case removal rate: with Kubernetes defaults, a correlated probe failure kills every affected pod in about 30 seconds (3 failures × 10-second period); your eviction budget is whatever survives that. Second, detection floor: an active checker cannot distinguish dead from slow faster than its timeout, and the Kubernetes default timeout is 1 second, which is why CPU throttling reads as death. Neither number appears in any dashboard by default.
Configuration defaults were read from the projects' current source trees on 2026-08-31 and can drift; incident figures are the operators' own reported numbers, not independent measurements. This page was researched from an environment that could fully fetch GitHub-hosted sources but had to verify web-hosted postmortems through live search results rather than full page fetches; the ledger (sources.md, shipped beside this page) marks which is which, row by row.
Every source behind this page, graded. Postmortems and source code outweigh commentary; vendor documentation is labelled as such. Filter by kind.
The definitive recent instance of a health checker removing healthy capacity: NLB checks flapped during recovery, pulling nodes and targets from DNS repeatedly.
Mass health-check failures against healthy instances during AWS network saturation; HAProxy's panic mode ignored the checks and kept balancing across all backends.
HAProxy instances ran for hours with stale backend state; when autoscaling removed the old backends they still pointed at, the map failed before any server did.
A 43-second partition, a correct quorum decision, an automated cross-country promotion, and 24 hours of reconciliation because writes existed on both sides.
The canonical gray-failure outage: contention made Consul servers slow rather than dead, health checks kept passing, and leadership flapped among defective leaders for days.
A mandatory fail-closed checker crash-looped on bad global data; every API request it could not evaluate was rejected. Remediations: fail open, feature flags, staged data propagation.
The budget as shipped defaults: 10% max ejection per cluster, 50% panic floor, ejection backoff with optional return jitter "to prevent a 'thundering herd' effect".
The eviction budget, proposed for the kubelet (honour PodDisruptionBudget before acting on probe failures), discussed, labelled a feature, and frozen. Kubernetes' missing gate has an eight-year-old paper trail.
1-second timeout, 10-second period, 3 consecutive failures. The 1-second default is the load-bearing surprise: a CPU-throttled pod misses it while doing useful work.
A production phi accrual implementation, header citing Hayashibara, convicting when the scaled phi crosses phi_convict_threshold (default 8, "most users should never need to adjust this").
Lifeguard in production Go: an "awareness" score of the local node's own health, used to scale probe timeouts so a sick observer accuses fewer healthy peers.
When renewals fall below renewalPercentThreshold (default 0.85), the registry "perceives this as a danger and stops expiring instances", per the javadoc; mass silence is read as a network event, not mass death.
The community thread on what the budget feels like from inside: ejection proceeds until the cap, and past the panic threshold requests fail together rather than one host at a time.
The design record for the check that kills slow starters, with both prior workarounds rejected for stated reasons: initialDelaySeconds "delays deadlock detection", a high failureThreshold forfeits timely kills after startup.
Motivated by a reported outage: an ingress controller with a 3600-second drain grace wedged, and the liveness kill waited the full hour, "the worst possible outcome".
Mark Roth's design for clients consuming the server's self-reported health, with the motivating case spelled out: a server that is up while a dependency "is not available", and alternatives considered in the record.
The clearest operator statement of the core tension: thorough checks catch more, and "the harm done by a false positive failure across the entire fleet" caps how much you may act on them. Names fail-open and the black-hole effect.
The self-report pattern: a draining backend keeps serving but broadcasts "stop sending", reaching all clients in one or two round trips instead of a probe interval.
The practitioner mechanics of probe-induced outages, including the sharpest sentence in the folklore: liveness plus an external dependency means "a single database hiccup will restart all containers".
The post that made the danger common knowledge: most workloads need no liveness probe at all, and a wrong one subtracts availability.
A firsthand correlated-false-positive outage: readiness checks on an auth dependency removed every pod from the load balancer simultaneously.
Field notes on the budget misfiring in the small: with tiny endpoint pools and aggressive retries, ejection storms brown out a mesh service even inside the 10% cap.
Defines differential observability from Azure incident data: detectors and applications systematically disagree about health, and the disagreement is where major outages live.
Replaces quadratic heartbeating with random probing, indirect probes, and a suspicion state before conviction; the ancestor of Consul, Serf and Uber's Ringpop membership.
Suspicion as a continuous value on "a scale that is dynamically adjusted to reflect current network conditions", decoupling detection from any fixed timeout.
Extends SWIM with the observer's self-suspicion; evaluation shows false positives cut by more than 98% at a setting that leaves median detection latency unchanged.
The 2017 paper's authors report how Azure operationalized differential observability, bridging "different components' perceptions of what constitutes failures". Slides published by USENIX.
Long-form independent walkthrough of the 2018 incident: what Orchestrator saw, what the automation did, and where a human in the loop would have changed the day.
The official reference, carrying its own warning: "Incorrect implementation of liveness probes can lead to cascading failures", a caution that entered the docs after user pressure (kubernetes/website#16607).
The standard SERVING / NOT_SERVING self-report service, letting servers "signal that they are not healthy without actually tearing down connections".
Six rungs from a toy checker to a detection plane you would trust. The line from toy to real is crossed at rung 4, where you give the detector a budget and then try to make it overspend.
Five worker processes behind a toy proxy; a checker that GETs /healthz every 2 s and evicts after 3 failures. Then make one worker slow (sleep 900 ms) rather than dead.
Done when: the slow-but-working worker gets evicted while serving successfully. Teaches: a timeout cannot tell dead from slow; every threshold is a latency assumption.
Replace the counter with phi accrual: track inter-arrival times of successful checks, emit phi, convict at a threshold. Port the logic from Cassandra's FailureDetector.java.
Done when: the same slow worker's phi rises and falls without conviction, and a killed worker convicts within seconds. Teaches: why phi 8 is a different kind of number from "3 failures".
Feed real request outcomes into the same suspicion score (consecutive errors, success-rate deviation), Envoy-outlier style. Compare detection latency against active checks alone.
Done when: a worker returning 500s is caught before its next active check would have run. Teaches: passive detection is faster and free, and blind at zero traffic.
Add the gate: at most one eviction per 30 s, at most 40% of the pool evicted, fail open past that. Now block the shared dependency all five workers check, and watch.
Done when: the correlated failure evicts one worker, then the gate locks and traffic keeps flowing; without the gate, the same event empties the pool. Teaches: the single design difference between Slack's January 2021 and an empty backend list.
Give workers a NOT_SERVING state and a drain endpoint; deploy by draining before stopping. Measure request errors during a rolling restart, before and after.
Done when: a full rolling restart completes with zero failed requests. Teaches: planned exits should be announced, not detected; the SRE book's 1–2 RTT claim, reproduced.
The Lifeguard scenario: CPU-starve the checker process itself (or add 500 ms of jitter to its network) and count false convictions over an hour, with and without scaling its own timeouts by a self-health score.
Done when: self-suspicion cuts false convictions by an order of magnitude, and you can say which component of your production stack plays the checker role and what happens when it is the sick one. Teaches: the AWS October 2025 mechanism, at desk scale.
The queries that actually found this material, grouped by what they surface. The vocabulary is the value: each term below was learned from one source and unlocked the next.
"health check" postmortem "removed capacity" OR "removed from DNS""panic mode" OR "fail open" health check outage postmortem"liveness probe" cascading restart production "we""velocity control" NLB health check AZ failoversite:github.com issue "livenessProbe" mass failures PodDisruptionBudgetsite:github.com envoy outlier detection panic threshold issuekubernetes KEP startupProbe "alternatives" liveness holdoffgrpc proposal "client-side health checking" A17"phi accrual failure detector" convict threshold cassandra"gray failure" "differential observability" azurelifeguard swim "local health" false positives arxivSWIM "infection-style" membership suspicion DSN 2002max_ejection_percent default site:github.com envoy protorenewalPercentThreshold eureka self preservationphi_convict_threshold cassandra.yaml default"healthy_panic_threshold" envoy runtime default 50