Evidence ledger
30 sources
Checked 31 Aug 2026
Evidence ledger
One row per claim in Deciding a server is dead: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how production systems decide a peer is dead (health checks, probes, failure detectors), and why the verdict-and-action loop itself keeps causing the outage.
Research date: 2026-08-31. All sources checked 2026-08-31.
Access note for this session: the research environment's egress proxy allowed direct fetches only of github.com (git protocol), raw.githubusercontent.com, hashicorp.com and cloud.google.com. Rows marked fetched were retrieved in full (raw file or shallow clone) and quotes are copied verbatim from the fetched bytes. Rows marked search-verified were confirmed to exist via live web-search results on the research date; the supporting quote or figure is copied from the search result content for that page, not from a full page fetch. No row is cited from memory.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Envoy | Panic threshold (arch overview, docs source) | vendor | current docs (fetched from main) |
2026-08-31 fetched | https://github.com/envoyproxy/envoy/blob/main/docs/root/intro/arch_overview/upstream/load_balancing/panic_threshold.rst | Envoy disregards health status when too few hosts are available; default 50% | "if the percentage of available hosts in the cluster becomes too low, Envoy will disregard health status and balance either amongst all hosts or no hosts... The default panic threshold is 50%." |
| 2 | Envoy | Panic threshold (same doc) | vendor | current docs | 2026-08-31 fetched | https://github.com/envoyproxy/envoy/blob/main/docs/root/intro/arch_overview/upstream/load_balancing/panic_threshold.rst | The stated purpose is stopping cascade | "The panic threshold is used to avoid a situation in which host failures cascade throughout the cluster as load increases." |
| 3 | Envoy | Panic threshold (same doc) | vendor | current docs | 2026-08-31 fetched | https://github.com/envoyproxy/envoy/blob/main/docs/root/intro/arch_overview/upstream/load_balancing/panic_threshold.rst | Fail-open vs fail-closed is configurable, and the right choice depends on the upstream's failure shape | "Choosing to fail traffic during panic scenarios can help avoid overwhelming potentially failing upstream services... This may be a good tradeoff to make if a given service is observed to fail in an all-or-nothing pattern." |
| 4 | Envoy | outlier_detection.proto |
source | current (fetched from main) |
2026-08-31 fetched | https://github.com/envoyproxy/envoy/blob/main/api/envoy/config/cluster/v3/outlier_detection.proto | Passive detection may evict at most 10% of a cluster by default | "The maximum % of an upstream cluster that can be ejected due to outlier detection. Defaults to 10%." |
| 5 | Envoy | outlier_detection.proto |
source | current | 2026-08-31 fetched | https://github.com/envoyproxy/envoy/blob/main/api/envoy/config/cluster/v3/outlier_detection.proto | Ejection-time jitter exists specifically to prevent synchronized return | "The maximum amount of jitter to add to the ejection time, in order to prevent a 'thundering herd' effect where all proxies try to reconnect to host at the same time. Defaults to 0s." |
| 6 | Envoy | Issue #17650, outlier detection vs admission control | source | 2021 (issue) | 2026-08-31 search-verified | https://github.com/envoyproxy/envoy/issues/17650 | Practitioners hit the interaction between ejection and panic: eject to the cap, then everything fails at once | Search-result summary of thread: "outlier detection will eject hosts until the panic threshold and then all requests will fail immediately." |
| 7 | Apache Cassandra | FailureDetector.java (trunk) |
source | current (fetched from trunk) | 2026-08-31 fetched | https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/gms/FailureDetector.java | Cassandra implements the phi accrual detector and convicts on a continuous suspicion score | File header cites "'The Phi Accrual Failure Detector' by Hayashibara"; conviction code: if (PHI_FACTOR * phi > getPhiConvictThreshold()) ... listener.convict(ep, phi); |
| 8 | Apache Cassandra | cassandra.yaml (trunk) |
source | current | 2026-08-31 fetched | https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml | Default conviction threshold is phi = 8 | "# phi value that must be reached for a host to be marked down. # most users should never need to adjust this. # phi_convict_threshold: 8" |
| 9 | Kubernetes | types.go probe API (staging, master) |
source | current | 2026-08-31 fetched | https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/api/core/v1/types.go | Default probe: 1s timeout, 10s period, 3 consecutive failures | "Number of seconds after which the probe times out. Defaults to 1 second."; "How often (in seconds) to perform the probe. Default to 10 seconds."; "Minimum consecutive failures... Defaults to 3." |
| 10 | Kubernetes | Docs: Liveness, Readiness, and Startup Probes (source md) | vendor | current | 2026-08-31 fetched | https://kubernetes.io/docs/concepts/workloads/pods/probes/ | The project's own docs warn the mechanism can cascade | "Incorrect implementation of liveness probes can lead to cascading failures." (fetched from kubernetes/website content/en/docs/concepts/workloads/pods/probes.md) |
| 11 | Kubernetes | Issue #66230: Prevent mass livenessProbe failures from taking down all pods in a Deployment | source | 2018-07, still open, lifecycle/frozen | 2026-08-31 search-verified | https://github.com/kubernetes/kubernetes/issues/66230 | The eviction-budget idea was proposed for kubelet probes and never implemented | Search-verified: issue proposes to "honor the maxUnavailable constraint of the PodDisruptionBudget and stop further liveness probe actions once the constraint is violated"; labeled kind/feature, lifecycle/frozen. |
| 12 | Kubernetes | Website issue #16607: Liveness Probes: mention that they can worsen app availability | source | 2019 | 2026-08-31 search-verified | https://github.com/kubernetes/website/issues/16607 | The docs caution exists because users argued for it after incidents | Issue title as returned by search: "Liveness Probes: mention that they can worsen app availability". |
| 13 | Kubernetes | KEP-950: pod-startup liveness-probe holdoff (startupProbe) |
adr | v1.16–1.20 (fetched) | 2026-08-31 fetched | https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/950-liveness-probe-holdoff/README.md | startupProbe exists because both prior workarounds were rejected with stated reasons | "Slow starting containers... are either killed before being up, or could be left deadlocked during a very long time before being killed." Alternatives section rejects initialDelaySeconds ("delays deadlock detection") and high failureThreshold ("prevents the container from being killed in a timely manner if it deadlocks"). |
| 14 | Kubernetes | KEP-2238: probe-level terminationGracePeriodSeconds | adr | v1.21 era (fetched) | 2026-08-31 fetched | https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2238-liveness-probe-grace-period/README.md | A real operator outage motivated decoupling probe kill from drain grace | Quoted in KEP from k8s issue #64715: "the controller uses a 3600s grace period so that it can drain long requests... but when the controller wedges it takes an hour to resolve, which is the worst possible outcome." |
| 15 | HashiCorp | memberlist README + awareness.go |
source | current (shallow clone) | 2026-08-31 fetched | https://github.com/hashicorp/memberlist | Lifeguard is implemented: the detector scores its own health and slows itself down when sick | README: "Another set of extensions, that we call Lifeguard, are made to make memberlist more robust in the presence of slow message processing (due to factors such as CPU starvation, and network delay or loss)." awareness.go: "awareness manages a simple metric for tracking the estimated health of the local node"; ScaleTimeout scales probe timeouts by the score. |
| 16 | HashiCorp | Lifeguard: Local Health Awareness for More Accurate Failure Detection (Dadgar, Phillips, Currey) | paper | 2017-07 (arXiv 1707.00788) | 2026-08-31 search-verified | https://arxiv.org/abs/1707.00788 | Considering "maybe I am the sick one" cuts false positives by ~50x | Search-verified abstract/eval figures: "with β=6, median latencies remain at their SWIM levels, but false positives are reduced by over 98% (more than 50x)". |
| 17 | Netflix | Eureka AbstractInstanceRegistry.java + DefaultEurekaServerConfig.java |
source | current (shallow clone) | 2026-08-31 fetched | https://github.com/Netflix/eureka | Self-preservation: below 85% renewal rate the registry stops trusting missed heartbeats entirely | Javadoc: when renewals drop below getRenewalPercentThreshold(), "eureka perceives this as a danger and stops expiring instances." Default: renewalPercentThreshold, 0.85. |
| 18 | Cornell / Das, Gupta, Motivala | SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol (DSN 2002) | paper | 2002 | 2026-08-31 search-verified | https://www.cs.cornell.edu/projects/Quicksilver/public_pdfs/SWIM.pdf | All-to-all heartbeating does not scale; suspicion before conviction is load-bearing | Search-verified: motivated by "the unscalability of traditional heart-beating protocols, which either impose network loads that grow quadratically with group size, or compromise response times or false positive frequency". |
| 19 | Hayashibara et al. | The phi accrual failure detector (SRDS 2004) | paper | 2004-10 | 2026-08-31 search-verified | https://www.semanticscholar.org/paper/The-%CF%86-Accrual-Failure-Detector-Hayashibara-D%C3%A9fago/65c40c79e30c1ef33c97a22a3d52cc9f0415a477 | Suspicion as a continuous, network-adjusted value instead of a binary timeout | Search-verified: "the basic idea of the φ failure detector is to express the value of φ on a scale that is dynamically adjusted to reflect current network conditions"; SRDS 2004, pp. 66–78. |
| 20 | Microsoft Research | Gray Failure: The Achilles' Heel of Cloud-Scale Systems (HotOS 2017) | paper | 2017-05 | 2026-08-31 search-verified | https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf | Differential observability: the detector and the application disagree about health | Search-verified: "A key feature of gray failure is differential observability: that the system's failure detectors may not notice problems even when applications are afflicted by them." |
| 21 | USENIX SREcon24 Americas | Gray Failure talk, Ryan Huang (U. Michigan) and Ze Li (Microsoft Azure) | talk | 2024-03-20 | 2026-08-31 search-verified | https://www.usenix.org/conference/srecon24americas/presentation/li | Azure operationalized differential observability; slides published | Talk page abstract (search-verified): experiences with gray failure in Microsoft Azure, "bridged the gap between different components' perceptions of what constitutes failures". Slides: https://www.usenix.org/system/files/srecon24americas_slides-li.pdf |
| 22 | HashiCorp (Armon Dadgar) | SWIM protocol talk deck | talk | n.d. (deck) | 2026-08-31 search-verified | https://speakerdeck.com/armon/swim-scalable-weakly-consistent-infection-style-process-group-membership-protocol | The SWIM lineage into Serf/Consul is presented by its implementer | Deck exists under Dadgar's account; title matches the DSN 2002 paper it presents. |
| 23 | AWS | Summary of the Amazon DynamoDB Service Disruption in US-EAST-1, October 19–20, 2025 | postmortem | 2025-10 | 2026-08-31 search-verified | https://aws.amazon.com/message/101925/ | NLB health checks flapped and removed healthy capacity; fix is a removal-rate budget | Search-verified quotes: "Health checks alternated between failing and healthy, which caused NLB nodes and backend targets to be removed from DNS, only to be returned to service when the next health check succeeded." Remediation: "adding a velocity control mechanism to limit the capacity a single NLB can remove when health check failures cause AZ failover." Timeline: 11:48 p.m. PDT Oct 19 to 2:20 p.m. PDT Oct 20. |
| 24 | ThousandEyes | AWS Outage Analysis: October 20, 2025 | blog | 2025-10 | 2026-08-31 search-verified | https://www.thousandeyes.com/blog/aws-outage-analysis-october-20-2025 | Independent external measurement of the same incident phases | Search-verified: NLB health check failures produced connection errors during recovery; DynamoDB endpoint resolution failed from ~11:48 p.m. PDT. |
| 25 | Slack | Slack's Outage on January 4th 2021 | postmortem | 2021-02 | 2026-08-31 search-verified | https://slack.engineering/slacks-outage-on-january-4th-2021/ | Fail-open saved them: panic mode ignored mass health-check failures during network saturation | Search-verified: load balancers "have a feature called 'panic mode' which balances requests across all instances when many are failing health checks"; panic mode "plus retries and circuit breaking" restored degraded service. |
| 26 | Slack | A Terrible, Horrible, No-Good, Very Bad Day at Slack (May 12, 2020 incident) | postmortem | 2020 | 2026-08-31 search-verified | https://slack.engineering/a-terrible-horrible-no-good-very-bad-day-at-slack/ | The liveness map itself goes stale: HAProxy instances held old backend state and ran out of usable backends | Search-verified: "most HAProxy instances were only able to send requests to older webapp backends", instances "more than eight hours old with stale backend state"; outage triggered when autoscaling terminated older instances. |
| 27 | GitHub | October 21 post-incident analysis | postmortem | 2018-10-30 | 2026-08-31 search-verified | https://github.blog/news-insights/company-news/oct21-post-incident-analysis/ | A correct dead verdict with an irreversible action: 43s partition, automated failover, 24h11m degradation | Search-verified: connectivity lost for 43 seconds; Orchestrator "began a process of leadership deselection" and failed clusters over to US West; degraded service for "24 hours and 11 minutes". |
| 28 | Roblox | Roblox Return to Service 10/28–10/31 2021 | postmortem | 2022-01 | 2026-08-31 search-verified | https://about.roblox.com/newsroom/2022/01/roblox-return-to-service-10-28-10-31-2021 | Nodes were alive-but-slow (gray failure); leadership kept flapping among "healthy" leaders; 73 hours | Search-verified: 73-hour outage; Consul streaming "exacerbates the amount of contention on a single Go channel, which causes blocking during writes"; team "made the pragmatic decision to prevent the problematic leaders from staying elected". |
| 29 | Google Cloud | Incident report, June 12 2025 Service Control outage | postmortem | 2025-06 | 2026-08-31 search-verified | https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW | A fail-closed checker on the request path crash-looped globally; remediation is to fail open | Search-verified via multiple analyses of the official report: null pointer in Service Control quota checking crash-looped binaries globally (~7h27m total, red-button rollout in ~40 minutes); remediation includes "modularizing Service Control's architecture to fail open rather than closed" and "enforcing feature flag protection". |
| 30 | AWS (David Yanacek) | Implementing health checks, Amazon Builders' Library | blog | 2019-12 | 2026-08-31 search-verified | https://aws.amazon.com/builders-library/implementing-health-checks/ | The central tension, fail-open, and the black-hole effect are all named by an operator | Search-verified: tension "between the benefits of thorough health checks... and the harm done by a false positive failure across the entire fleet"; fail-open in NLB, ALB, Route 53; a fast-failing server "creating a 'black hole' in the service fleet by attracting more requests than healthy servers". |
| 31 | SRE book, ch. 20, Load Balancing in the Datacenter (lame duck) | blog | 2016 | 2026-08-31 search-verified | https://sre.google/sre-book/load-balancing-datacenter/ | Self-reported "leaving" state beats external detection for planned exits | Search-verified: "the backend task is listening on its port and can serve, but is explicitly asking clients to stop sending requests"; lame-duck fact propagates to clients "typically in 1 or 2 RTT". | |
| 32 | gRPC | A17: Client-Side Health Checking (Mark D. Roth) | adr | 2018-08-27 | 2026-08-31 fetched | https://github.com/grpc/proposal/blob/master/A17-client-side-health-checking.md | Health is an application-level protocol with explicit design alternatives, not a TCP property | Fetched: proposal for "application-level health-checking on the client side"; background: server signals unhealthy "when (e.g.) a server is itself up but another service that it depends on is not available." |
| 33 | gRPC | Health Checking Protocol (health-checking.md) |
vendor | current | 2026-08-31 fetched | https://github.com/grpc/grpc/blob/master/doc/health-checking.md | The standard server-side health protocol: SERVING / NOT_SERVING self-report | Fetched from grpc/grpc doc/health-checking.md. |
| 34 | Colin Breck | Kubernetes Liveness and Readiness Probes: How to Avoid Shooting Yourself in the Foot | blog | 2019 (plus 2021 follow-up) | 2026-08-31 search-verified | https://blog.colinbreck.com/kubernetes-liveness-and-readiness-probes-how-to-avoid-shooting-yourself-in-the-foot/ | Probes "can unintentionally reduce service availability"; dependency-checking liveness is the worst case | Search-verified: probes "can unintentionally reduce service availability, or result in prolonged outages"; liveness probes "in combination with an external dependency are the worst situation leading to cascading failures: a single database hiccup will restart all containers". |
| 35 | Henning Jacobs (Zalando) | Liveness Probes are Dangerous | blog | 2019 | 2026-08-31 search-verified | https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html | The practitioner-consensus title; do not check dependencies in liveness | Existence and thesis verified in search results (title and site); widely cited in the k8s probe literature. |
| 36 | Encore | Distributed Systems Horror Stories: Kubernetes Deep Health Checks | blog | 2023 | 2026-08-31 search-verified | https://encore.dev/blog/horror-stories-k8s | A deep readiness check on an auth dependency removed every pod at once | Search-verified: "The failure of the auth service leads to all of our pods being removed from the load balancer for our service; we have a complete outage." |
| 37 | Lorin Hochstein | Quick thoughts on the recent AWS outage (Surfing Complexity) | blog | 2025-10-25 | 2026-08-31 search-verified | https://surfingcomplexity.blog/2025/10/25/quick-thoughts-on-the-recent-aws-outage/ | Independent practitioner reading of the Oct 2025 event as an automation/feedback failure | Post exists at this URL and date per search results; commentary on the AWS October 2025 outage. |
| 38 | Michal Drozd | Envoy Outlier Detection Brownouts: When the Mesh Ejects Healthy Pods | blog | n.d. (recent) | 2026-08-31 search-verified | https://www.michal-drozd.com/en/blog/envoy-outlier-detection-brownouts/ | Outlier detection misfires in small pools and with aggressive retries; ejection storms brown the service out | Search-result summary: "outlier detection can become a brownout generator with small endpoint pools, aggressive retry policies, or cross-zone constraints." |
| 39 | HeyOnCall | Kubernetes Liveness Probes and CPU Limit Risks: Self-Reinforcing CrashLoopBackOff | blog | n.d. (recent) | 2026-08-31 search-verified | https://heyoncall.com/blog/kubernetes-liveness-probes-and-cpu-limits-risks-self-reinforcing-crashloopbackoff | The probe+throttle feedback loop, concretely: 1s default timeout under CPU throttling restarts a busy pod forever | Search-result summary: "As the process was busy and being throttled, HTTP liveness probes would hit their default 1-second timeout, causing kubelet to restart the Pod... triggering yet another restart." |
| 40 | The Downtime Project | GitHub's 43 Second Network Partition (podcast) | talk | 2021 | 2026-08-31 search-verified | https://downtimeproject.com/podcast/githubs-43-second-network-partition/ | Independent long-form analysis of the GitHub 2018 failover decision | Episode page exists per search; discusses the 43-second partition and Orchestrator failover. |