Evidence ledger
30 sources
Checked 16 Sep 2026
Evidence ledger
One row per claim in The router finds out last: taking a server out of service without dropping requests: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Tiers follow the skill's grading (postmortem, source, adr, casestudy,
blog, paper, talk, vendor). Checked date for every row: 2026-09-16.
Retrieval mode, disclosed. This session ran in a remote environment whose network egress
allows github.com (and its raw/gist hosts), gitlab.com, and storage.googleapis.com
directly. Rows marked [F] were fetched in full this session. Rows marked [S] are hosts
outside the egress allowlist; their content was retrieved this session through the search tool,
which returns page content and quotes server-side. For [S] rows the supporting text below is
what the search tool returned; where it could not be confirmed verbatim it is used in the guide
as an attributed report, not as a quotation. No row is cited from memory. Conference talks and
video platforms were not reachable at all in this environment; the talk tier is absent and the
guide says so explicitly.
| # | Org | Title | Tier | Published | Mode | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Kubernetes SIG Node | KEP-3960: Introducing Sleep Action for PreStop Hook | adr | 2023-04, GA v1.34 (2025) | [F] | https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3960-pod-lifecycle-sleep-action/README.md | Upstream made sleep a first-class API field because the exec-sleep workaround "requires a sleep binary in the image" |
"Another way to run sleep in a container is to use exec command in preStop hook like command: [\"/bin/sh\",\"-c\",\"sleep 20\"]. However, this requires a sleep binary in the image." |
| 2 | Kubernetes SIG Node | KEP-3960 (same document) | adr | 2023-04 | [F] | https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3960-pod-lifecycle-sleep-action/README.md | The sleep exists to lose the race deliberately: delay shutdown so endpoint removal wins | "PreStop hook starts, which will delay the shutdown sequence by 5 seconds. During this time, the Endpoint Controller will remove the terminating pod, and the traffic will be sent to other running pods." |
| 3 | Kubernetes SIG Network | KEP-1669: Proxy Terminating Endpoints | adr | 2020, stable v1.28 | [F] | https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/1669-proxy-terminating-endpoints/README.md | The traffic-loss window during rollouts is a race between pod termination and the LB's health-check probe interval | "Upon performing a rolling update of a Deployment, there can be a small window of time where old pods on a node are terminating (hence not \"Ready\") but the load balancer has not probed kube-proxy's healthCheckNodePort yet." Also: "The likelihood of this traffic loss is impacted by two factors: the number of local endpoints on the node and the interval between health checks from the load balancer." |
| 4 | Kubernetes SIG Network | KEP-1672: Tracking Terminating Endpoints | adr | 2020-07 (alpha v1.20) | [F] | https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/1672-tracking-terminating-endpoints/README.md | Before 1.20 the API erased terminating endpoints entirely; serving/terminating conditions had to be added so consumers could drain |
"terminating endpoints are considered 'not ready' regardless of their actual readiness"; new condition serving is "set based on pod readiness regardless of terminating state" |
| 5 | gRPC | gRFC A9: Server-side Connection Management | adr | 2017 | [F] | https://github.com/grpc/proposal/blob/master/A9-server-side-conn-mgt.md | Long-lived HTTP/2 connections defeat L4 load balancing, so servers rotate connections with GOAWAY | "with HTTP/2 and clients keeping idle connections open, they are not able to quickly distribute load away from hot nodes or to new nodes"; "the server should send GOAWAY with status code NO_ERROR ... outstanding RPCs should be permitted to complete." |
| 6 | Kubernetes | PR #97238: kube-proxy handle terminating endpoints (merged 2021-06-29) | source | 2021-06 | [F] | https://github.com/kubernetes/kubernetes/pull/97238 | kube-proxy now routes to terminating-but-serving pods as a last resort, explicitly to patch the kill-vs-notice race | "this change patches up a race condition between when a pod is killed and when the external load balancer notices that it has been killed"; "kube-proxy will send the traffic to the Terminating pod rather than dropping it" |
| 7 | Kubernetes | Issue #106476: Pod Termination handling kicks in before the ingress controller has had time to process | source | 2021-11, still open (lifecycle/frozen) | [F] | https://github.com/kubernetes/kubernetes/issues/106476 | The ask to sequence SIGTERM after LB deregistration is recorded, accepted for triage, and unimplemented since 2021 | "The SIGTERM should only start after the ingress controller/LB has removed the target from the target group."; "Readiness gates work pretty good for pod startup/rollout but lack support during pod deletion." |
| 8 | Google Cloud (ingress-gce users) | Issue #2222: Rolling restart results in 502 for ~minute from LoadBalancer | source | 2023-08, closed as not planned | [F] | https://github.com/kubernetes/ingress-gce/issues/2222 | With a cloud LB routing straight to pods, one replaced pod produces 15-60 s of 502s despite a 20 s preStop sleep and 120 s grace | "502 is returned even when only one pod is replaced - the second old pod is still alive, and yet LB returns 502." Reported impact: 15-60 seconds of 502 responses per rolling restart. |
| 9 | Linkerd | Issue #11084: Consistent 502s during shutdown/rolling a deployment | source | 2023-07, closed | [F] | https://github.com/linkerd/linkerd2/issues/11084 | A service mesh adds its own shutdown-ordering race: the app dies while the sidecar still accepts traffic | "I can consistently get 502s from the service during a deployments rolling restart"; logs show "error trying to connect: Connection refused (os error 111)" before the proxy received SIGTERM |
| 10 | AWS (aws-load-balancer-controller users) | Issue #1065: 400/502/504 errors while doing rollout restart or rolling update | source | 2019-11, closed | [F] | https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1065 | Measured error mix during an ALB-fronted rollout under 500 QPS load; preStop sleep alone did not eliminate it | "Code 400 : 13 (0.7 %) / Code 502 : 4 (0.2 %) / Code 504 : 1 (0.1 %)" under Fortio at 500 QPS during a rolling update, with 60 s terminationGracePeriodSeconds |
| 11 | Envoy | Issue #3804: RFC: Hot restart across hot restart versions with SO_REUSEPORT | source | 2018-07, open, unimplemented | [F] | https://github.com/envoyproxy/envoy/issues/3804 | Socket-handover upgrades have a recorded, unresolved design argument; the fallback is "a prolonged cluster drain for upgrades" | "There is no standard way to upgrade when the hot restart version changes... the only way to do this would be something like a prolonged cluster drain for upgrades." |
| 12 | Cloudflare | tableflip (Go library for zero-downtime upgrades) | source | 2018- | [F] | https://github.com/cloudflare/tableflip | Process-handover constraints as Cloudflare encodes them: complete replacement, init grace, crash-during-init is fine, one upgrade at a time | "No old code keeps running after a successful upgrade"; "The new process has a grace period for performing initialisation"; "Crashing during initialisation is OK"; "Only a single upgrade is ever run in parallel" |
| 13 | foriequal0 (open source) | pod-graceful-drain | source | 2021- | [F] | https://github.com/foriequal0/pod-graceful-drain | The webhook alternative to sleeping: intercept deletion and hold the pod until deregistration; the author's stated reason is that sleep is fragile and "ugly" | "intercepts the deletion/eviction of a pod ... to prevent the pod from getting terminated for a period"; "sometimes, 'sleep' command might not be available on some containers... And it is ugly." Tagline: "You don't need lifecycle: { preStop: { exec: { command: [\"sleep\", \"30\"] } } }" |
| 14 | GitHub | multibinder | source | 2016-12 | [F] | https://github.com/github/multibinder | GitHub's socket-inheritance implementation: LISTEN sockets held by a broker and passed over a UNIX socket so HAProxy reloads never close the port | "It accepts connections on a UNIX domain socket and binds an arbitrary number of LISTEN sockets"; "the LISTEN socket is sent over the UNIX domain socket using ancillary data. Subsequent identical binds receive the same LISTEN socket." |
| 15 | gRPC Java | Issue #11351: gRPC unavailable during services rollout when using client side LB | source | 2024-07, closed as not planned | [F] | https://github.com/grpc/grpc-java/issues/11351 | Client-side LB re-derives the same drain problem: 60 s preStop and 180 s grace still produced UNAVAILABLE during rollouts | "finishConnect(..) failed: Connection refused"; UNAVAILABLE errors persisted during rollouts despite 60-second preStop hooks and 180-second grace |
| 16 | Kubernetes | Pod Lifecycle documentation (termination of pods) | vendor | current (fetched from kubernetes/website main) | [F] | https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/ | The platform documents the race as intended behavior: endpoint removal is evaluated "at the same time as" the kubelet starts shutdown, with no ordering guarantee | "At the same time as the kubelet is starting graceful shutdown of the Pod, the control plane evaluates whether to remove that shutting-down Pod from EndpointSlice objects."; "By default, all deletes are graceful within 30 seconds."; preStop overrun gets "a small, one-off grace period extension of 2 seconds" |
| 17 | Envoy | Draining (architecture overview) | vendor | current | [F] | https://github.com/envoyproxy/envoy/blob/main/docs/root/intro/arch_overview/operations/draining.rst | Envoy's drain is protocol-aware: fail health checks, discourage traffic for a window, then GOAWAY / Connection: close per protocol | "The HTTP connection manager filter will add 'Connection: close' to HTTP1 requests, send HTTP2 GOAWAY, and terminate connections on request completion." |
| 18 | Envoy | Command line options: --drain-time-s, --parent-shutdown-time-s | vendor | current | [F] | https://github.com/envoyproxy/envoy/blob/main/docs/root/operations/cli.rst | Envoy's defaults size the drain window at 600 s and keep the old process 900 s | "--drain-time-s (default 600): The time in seconds that Envoy will drain connections during a hot restart..."; "--parent-shutdown-time-s (default 900)"; drain strategy gradual "increases to 100% as the drain time elapses" |
| 19 | AWS | aws-load-balancer-controller: annotations and pod readiness gates | vendor | current | [F] | https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/deploy/pod_readiness_gate.md | Readiness gates protect the registration side of a rollout only; the docs' motivating outage is a target group holding only Initial/Draining targets | "the target group might only have registered targets that are in »Initial« or »Draining« state; this results in service outage"; injected "via mutating webhook during pod creation" |
| 20 | AWS | Edit target group attributes (deregistration delay) | vendor | current | [S] | https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-target-group-attributes.html | The managed-LB drain default is 300 s, range 0-3600 s, and ends early only when connections are gone | "Elastic Load Balancing waits 300 seconds before completing the deregistration process"; "The range is 0-3600 seconds. The default value is 300 seconds."; "if a deregistering target has no in-flight requests and no active connections, Elastic Load Balancing immediately completes the deregistration" |
| 21 | Large-scale cluster management at Google with Borg (EuroSys 2015), §2.3 | paper | 2015 | [F] (PDF from storage.googleapis.com) | https://storage.googleapis.com/gweb-research2023-media/pubtools/pdf/43438.pdf | Graceful notice is best-effort even inside Google: SIGTERM-before-SIGKILL lands ~80% of the time | "Tasks can ask to be notified via a Unix SIGTERM signal before they are preempted by a SIGKILL, so they have time to clean up, save state, finish any currently-executing requests, and decline new ones... In practice, a notice is delivered about 80% of the time." | |
| 22 | Maglev: A Fast and Reliable Software Network Load Balancer (NSDI 2016), §5.2 context | paper | 2016 | [F] (PDF from storage.googleapis.com) | https://storage.googleapis.com/gweb-research2023-media/pubtools/pdf/44824.pdf | The load balancer layer itself is drained for upgrades, and the fleet-wide operation runs for over an hour | "when upgrading Maglevs in a cluster we do a rolling restart of machines, draining traffic from each one a few moments beforehand and restoring it once the Maglev starts serving again. This process may last over an hour" | |
| 23 | Candea & Fox (Stanford) | Crash-Only Software (HotOS IX) | paper | 2003 | [S] | https://www.usenix.org/conference/hotos-ix/crash-only-software | The opposing school: if crash recovery must work anyway, make crashing the only stop mechanism | "Crash-only programs crash safely and recover quickly. There is only one way to stop such software - by crashing it - and only one way to bring it up - by initiating recovery." (abstract, as returned by search retrieval; multiple independent results agree on the wording) |
| 24 | Google SRE | Site Reliability Engineering, ch. 20 "Load Balancing in the Datacenter" (lame duck state) | casestudy | 2016 | [S] | https://sre.google/sre-book/load-balancing-datacenter/ | The reference behavior: a stopping server keeps serving while actively telling clients to go away, and the state propagates to clients in 1-2 RTT | "The backend task is listening on its port and can serve, but is explicitly asking clients to stop sending requests."; allowing lame duck state "simplifies clean shutdown"; propagation "typically in 1 or 2 RTT" via active clients and UDP health checks |
| 25 | GitLab | Production incident #6736: Elevated error rates across GitLab.com WEB service (2022-03-31) | postmortem | 2022-03 | [F] | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6736 | A port-rename chart change reconfigured the cloud LB in place; 78 minutes of degradation across five windows; the corrective action was to build cluster-level draining | Root cause: chart port name changes "caused the load balancer in GCP to be re-configured" with timeouts while applying. Impact: 78 minutes total (9+33+8+7+21). Corrective action: "Implement graceful draining for entire regional clusters" |
| 26 | Slack | A Terrible, Horrible, No-Good, Very Bad Day at Slack (2020-05-12 outage) | postmortem | 2020 | [S] | https://slack.engineering/a-terrible-horrible-no-good-very-bad-day-at-slack/ | The membership-state machinery is its own failure domain: HAProxy slot exhaustion made LB state increasingly stale; the outage fired at the evening scale-down; 48 minutes down | HAProxy server-template slots exhausted after a morning scale-up; the state-management program exited early on "no empty slot", so "the list of backends in the HAProxy state became more and more stale"; outage 4:45-5:33 p.m. PDT, triggered when the webapp tier scaled down (as returned by search retrieval) |
| 27 | hyperbo.la (personal infra, Milo Turner) | Postmortem: 502s During Parameter Store Rollout | postmortem | n.d. (checked 2026-09-16) | [S] | https://hyperbo.la/w/secrets-in-parameter-store-postmortem/ | The tooling that replaces instances is part of the drain contract: an "unsafe ASG cycler script" cycled instances without safe draining and took the site to 100% 502s | Deploy of v0.149.2 "brought down the site and caused all requests to return 502"; root cause named as a misconfigured PrivateLink endpoint plus "an unsafe ASG cycler script" (as returned by search retrieval) |
| 28 | GitHub | GLB part 2: HAProxy zero-downtime, zero-delay reloads with multibinder | blog | 2016-12-01 | [S] | https://github.blog/news-insights/glb-part-2-haproxy-zero-downtime-zero-delay-reloads-with-multibinder/ | Before multibinder, GitHub used the standard HAProxy reload trick of dropping SYNs and letting clients retry; at their scale that window was customer-visible | At GitHub's scale "a customer-impacting number of connections" would hit the reload window; the prior standard solution was "dropping SYN packets during a small window, causing clients to retry the SYN packet shortly afterwards" (as returned by search retrieval) |
| 29 | Yelp | True Zero Downtime HAProxy Reloads | blog | 2015-04 | [S] | https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html | The pre-socket-handover state of the art: delay SYN packets with Linux qdiscs during the reload so the new process answers them | Yelp's technique delayed SYN packets "using Linux queuing disciplines (qdisc)", releasing queued packets when the qdisc was removed (as returned by search retrieval; technique corroborated by GitHub's GLB post and Envoy issue #3804 discussion, both [F]) |
| 30 | Slack | Migrating Millions of Concurrent Websockets to Envoy | blog | 2021-03-15 | [S] | https://slack.engineering/migrating-millions-of-concurrent-websockets-to-envoy/ | With hours-long connections, HAProxy reload-based config changes left old processes running "for many hours" while websockets drained; a fleet of half-retired processes | Slack "reloaded HAProxy, creating new processes to handle incoming connections while keeping the old process running for many hours to allow long-lived websocket connections to drain" (as returned by search retrieval) |
| 31 | learnk8s | Graceful shutdown and zero downtime deployments in Kubernetes | blog | current (site now learnkube.com) | [S] | https://learnkube.com/graceful-shutdown | The community's canonical writeup: endpoint propagation is eventually consistent and unsequenced against SIGTERM, so the standard fix is to wait before shutting down | Endpoint removal and SIGTERM "are not sequenced against each other"; propagation to kube-proxy, ingresses and cloud LBs takes real time, so the recommendation is to delay shutdown until routing has caught up (as returned by search retrieval) |
| 32 | Rakuten | Zero-Downtime Rolling Deployments in Kubernetes | blog | n.d. (checked 2026-09-16) | [S] | https://engineering.rakuten.today/post/graceful-k8s-delpoyments/ | An operator's statement of the race, independent of learnk8s: TERM delivery and endpoint-list removal "can happen in any order" | Kubernetes keeps sending new requests after TERM "because there is no orchestration between Kubernetes sending the TERM signal and removing the pod from its service endpoint list", the operations "can happen in any order with possible delays between them" (as returned by search retrieval) |
Absences worth recording
- Talks. No conference-talk tier appears above. Video platforms, InfoQ and conference sites were outside this environment's network allowlist, so no talk could be fetched and no timestamped claim could be verified. Talks on this topic exist (KubeCon and SREcon programmes list several); their absence here is an access limitation of this run, not evidence of absence.
- A postmortem that blames the cutover mechanism itself. As with the resharding dig in this repo, the incidents above blame stale membership state, bypassing tooling, or the control plane; none of the fetched incidents describes the drain sequence executing as designed and still causing the outage.
- Numbers for lame-duck adoption outside Google. No public figure was found for how many organisations implement subscription-style drain notification (as opposed to health-check polling or fixed sleeps).