Deciding a server is dead

Reconstructs the failure-detection plane behind health checks from six published incidents, five production codebases, four papers and the Kubernetes and gRPC design records. Shows that mature systems independently converged on one guardrail, a cap on what the detector may do before its verdicts are trusted, and gives the reader the decision table, defaults and failure classes to apply that rule to their own fleet.

What you get out of it

• Cap what the failure detector may do before you trust what it says: Envoy ejects at most 10% and ignores health below 50% availability, Eureka stops expiries below 85% renewals, AWS is adding NLB velocity control after October 2025.
• Correlated false positives, not missed deaths, dominate the published incident record; no postmortem in this corpus blames an undetected dead server.
• A check whose failure gates a kill must never touch dependencies: the deep-check cascade (Encore, k8s issue #66230, Breck) removes the whole fleet in one probe window.
• A correct verdict can still be the outage when the action is irreversible: GitHub's 43-second partition triggered an automated failover that took 24 hours to unwind.
• The detector should suspect itself: Lifeguard's self-health scoring cut false dead verdicts by more than 50x, and the AWS October 2025 flapping was a degraded checker, not degraded targets.

Scope

Why this, now. AWS's October 2025 us-east-1 postmortem made the pattern current again: its NLB remediation adds in 2025 the same removal budget Envoy has shipped as a default for years and Eureka has carried since 2012.

What it does not cover. Human-facing monitoring and alerting, consensus-protocol correctness, and client-side reaction to failure (retries, backoff, metastable overload), which has its own dig in this collection.