Every deploy deliberately retires servers that are still being sent work. The signal that kills the process and the update that stops traffic travel on different paths, at different speeds, with no acknowledgement between them. This guide reconstructs how Google, Kubernetes, AWS, Envoy, GitHub and Slack coordinate that removal, what broke when they did not, and why the most widely deployed fix in the industry is a timed pause.
Removing a server on purpose is a three-party coordination problem, and the three parties do not share a transaction.
State the problem without naming a technology: a fleet serves requests behind a routing layer that picks which server gets each one. Some other system, a deploy pipeline, an autoscaler, an operator running maintenance, decides a particular server must go. Three parties now have to agree: the thing that decides, the process that holds in-flight work, and the routing layer that keeps choosing backends from a membership list it believes. Termination is delivered to the process directly, in microseconds. The membership update is replicated out to every proxy, kernel table and client that routes, in seconds to minutes. Nothing sequences the two. Every dropped-request-on-deploy incident in this guide is that gap, observed from a different angle.
This is not an exotic failure. It runs on every rolling deploy, every scale-down and every node drain, which is why the error rate it produces is small, periodic and easy to normalise. The Kubernetes documentation states the race as designed behavior: endpoint removal is evaluated “at the same time as the kubelet is starting graceful shutdown”, with no ordering guarantee. Rakuten's engineering blog says it plainly: there is no orchestration between sending the termination signal and removing the pod from the endpoint list; the two can happen in any order.
The surprise this dig turned up sits at the end of that story. After a decade in which
practically every Kubernetes team independently rediscovered the same workaround, a
sleep in the preStop hook, the project made the workaround a first-class API
field: KEP-3960
shipped a native sleep action, stable in v1.34 (2025), partly because hardened
images had no sleep binary to exec. The platform's durable answer to a distributed race is a
timer, and that is not an accident or an embarrassment; sections 2 and 3 argue it is the
honest design once you refuse to put a synchronous barrier in the removal path. The second
surprise is Borg reporting that the graceful path is only best-effort even inside Google:
the SIGTERM notice arrives about 80% of the time. Whatever you build here is a latency
optimisation layered on crash-safety, never a substitute for it.
Scope: this guide covers the deliberate removal path, deploys, scale-downs, drains and proxy reloads, for stateless request-serving fleets. It does not cover deciding that an unresponsive peer is dead (the detection problem is a separate guide in this collection), nor stateful failover, nor draining batch workers from a queue.
Five systems that published their removal path converge on one sequence. They differ on exactly one axis: how the waiter learns it is safe to stop.
Google's RPC stack, Envoy, the AWS managed load balancers, Kubernetes and the gRPC
protocol each solve this with what is recognisably the same five-step sequence, assembled
from different parts. First, an intent signal reaches the server: SIGTERM in Borg and
Kubernetes, an admin drain endpoint in Envoy, a deregistration call at AWS. Second, the
server stops being selectable while it is still serving. Google's SRE book names this state
the lame duck: “the
backend task is listening on its port and can serve, but is explicitly asking clients to
stop sending requests”. Kubernetes expresses the same state through the API since
KEP-1672: a terminating endpoint stays visible with serving=true,
ready=false. An AWS target enters draining. Third, the removal
propagates through the routing plane. Fourth, in-flight requests finish, and persistent
connections are told to leave at the protocol layer, HTTP/2 GOAWAY or
Connection: close, because a connection that outlives the drain window pins
its client to the dying server. Fifth, a hard deadline fires regardless: SIGKILL when the
grace period expires, the deregistration delay elapsing, Envoy's
--parent-shutdown-time-s killing the old process at 900 seconds. Every
published implementation has all five steps; none skips the deadline.
The divergence point is step 3, and it divides the field into three designs. Google closes the loop: a task entering lame duck broadcasts the fact to its active clients, and because inactive clients still send periodic UDP health checks, the SRE book reports the state propagating in one or two round trips. The subscription is why Google can drain in seconds. Health-check-driven systems leave the loop half open: nothing announces the removal, the routing plane discovers it on its next probe, so the window is the probe interval times the unhealthy threshold, plus programming time. KEP-1669 identifies exactly this pair of factors for the traffic-loss window on Kubernetes services with local traffic policy. And the third design does not close the loop at all: it waits a fixed time and assumes propagation finished. That is the preStop sleep, the ALB deregistration delay, Envoy's 600-second drain window. I am calling this third design the open-loop wait; the sources call it endpoint propagation delay, deregistration delay, or nothing at all.
It is worth being precise about why the open loop won in the Kubernetes ecosystem, because the paper trail is unusually complete. The ask to close the loop exists: issue #106476 requests that “the SIGTERM should only start after the ingress controller/LB has removed the target from the target group”, notes that readiness gates “lack support during pod deletion”, and has sat accepted-but-frozen since November 2021. What merged instead, across five years, is a set of mechanisms that soften the race without sequencing it: KEP-1672 keeps terminating endpoints visible in the API (v1.20), PR #97238 lets kube-proxy fall back to a terminating pod rather than drop the connection, explicitly described by its author as patching “a race condition between when a pod is killed and when the external load balancer notices” (v1.22 era, merged June 2021), and KEP-3960 makes the wait itself a primitive (v1.34). The record does not state why sequencing was rejected; my reading, and it is a reconstruction, is that a synchronous barrier on pod deletion would hand every ingress controller and cloud LB a veto over the kubelet's termination path, and the project consistently chose eventual consistency over that coupling. Third-party projects fill the gap from outside: pod-graceful-drain uses an admission webhook to intercept the deletion and hold the pod until deregistration, its README arguing that the sleep is fragile on containers with no shell “and it is ugly”.
The state that says serving-but-leaving. Broadcast to clients at Google; expressed as
serving=true, terminating=true in the Kubernetes API; the draining
target state at AWS; failing health checks on purpose in Envoy's drain sequence.
Runs this way at: Google, Kubernetes, AWS
Sized by how the loop closes: 1–2 RTT when clients subscribe, probe interval times threshold when they poll, a configured constant when nobody tells anybody. The constant is load-bearing: too short drops requests, too long stalls every deploy.
Runs this way at: Kubernetes, Envoy
Graceful never means unbounded. SIGKILL after 30s by default on Kubernetes, 900s parent shutdown in Envoy, deregistration completing at 300s on an ALB whether or not connections finished. Borg's 80% figure says the deadline path runs constantly in production, not rarely.
Runs this way at: Kubernetes, Google Borg
Four forks, each with a recorded argument, and the condition that flips each one.
Connection: close / GOAWAY during its drain window| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Where the wait lives | Platform pause before SIGTERM | Barrier on deletion awaiting LB ack | A barrier gives every LB controller a veto over termination; eventual consistency was kept | #106476, KEP-3960 |
| Traffic to terminating pods | Route to serving-but-terminating as fallback | Drop on the floor | Dropping guarantees the error; the terminating pod usually still serves | PR #97238, KEP-1669 |
| Persistent connections | Server-initiated GOAWAY + max age | Wait for natural close | L4 balancers cannot move load off a connection that never closes | gRFC A9 |
| Shutdown philosophy | Crash-safe core, drain on top | Drain as the correctness mechanism | The notice is best-effort; one in five Borg terminations gets none | Borg §2.3, Candea & Fox |
| Proxy-tier reloads | Socket inheritance across processes | Drop or delay SYNs in the window | At large scale every reload window catches customer connections | multibinder, GLB part 2 |
| Sizing the LB drain delay | Bound it near p99 request duration | Keep the 300s default | The delay is a floor on deploy speed; AWS ends it early only when connections are gone | AWS ELB docs |
Decide explicitly which of the three loop designs you are running: subscription, polling, or open-loop timer. Most outages in section 4 happened to teams that believed they were running one design while actually running another; the sleep that everyone copies is correct exactly when its constant is derived from a measured propagation tail, and folklore when it is copied from a blog post.
The published record sorts into three failure classes, and none of them is the drain sequence itself misfiring.
Class one: the rumor travels slower than the kill. This is the base race of figure 1 observed live, and it is the most-reported class by far, three independent trackers in this corpus alone. Class two: the membership ledger itself is wrong, so no amount of waiting helps; Slack's May 2020 outage and GitLab's March 2022 incident are both here, and both were triggered by routine capacity operations, not by deploys. Class three: the removal tooling bypasses the drain contract entirely; the machinery existed, and the script did not use it. That the mechanism executing as designed appears nowhere in the incident record as a cause is itself a finding: the drain sequence is not where the risk lives. The risk lives in the state it depends on and the tooling that is supposed to invoke it.
Every value below is from the linked source; nothing is an estimate unless marked as one.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| SIGTERM notice delivery rate | ~80% | Google Borg | Share of task terminations that get the graceful notice before SIGKILL | 2015 | Borg paper §2.3 |
| Lame-duck propagation | 1–2 RTT | Broadcast to active clients; idle clients learn via UDP health checks | 2016 | SRE book ch. 20 | |
| Default termination grace | 30 s | Kubernetes | SIGTERM to SIGKILL; preStop overrun gets a one-off 2 s extension | 2026 | Pod lifecycle docs |
| LB deregistration delay, default | 300 s | AWS ALB | Range 0–3600 s; ends early only when no in-flight requests or active connections remain | 2026 | AWS ELB docs |
| Proxy drain window, default | 600 s | Envoy | --drain-time-s; gradual strategy ramps drain pressure to 100% across the window | 2026 | Envoy CLI docs |
| Old-process lifetime on hot restart | 900 s | Envoy | --parent-shutdown-time-s, the deadline behind the drain | 2026 | Envoy CLI docs |
| 502 window per rolling restart | 15–60 s | GCP NEG user | One pod replaced, 20 s preStop sleep, readiness passing | 2023 | ingress-gce #2222 |
| Rollout error mix under load | 0.7 / 0.2 / 0.1% | ALB user | HTTP 400 / 502 / 504 shares at 500 QPS during one rolling update | 2019 | alb-controller #1065 |
| Membership-state outage | 48 min | Slack | Stale HAProxy slots; triggered by the evening scale-down | 2020 | Slack postmortem |
| LB-reconfiguration incident | 78 min | GitLab.com | Five windows of elevated web errors from an in-place LB reconfig | 2022 | GitLab #6736 |
| LB-fleet rolling restart | >1 hour | Google Maglev | Whole-cluster upgrade of the load balancers themselves, each drained beforehand | 2016 | Maglev paper |
| Websocket drain via old processes | many hours | Slack | Pre-Envoy: each HAProxy reload left the old process running until websockets closed | 2021 | Slack engineering |
What to plan against: the deploy-time cost of removal is roughly the wait per instance times fleet size divided by rollout parallelism. With the ALB default of 300 seconds and serial batches, that is five minutes per batch before any application time; this is why the sizing decision in section 3 matters and why several sources cut the delay to 30 seconds or near their p99 request duration. That arithmetic is mine (derived, not reported). The measured figures above split cleanly: the Google numbers are primary measurements at scale; the 15–60 s and error-percentage figures are single-environment reports, valuable for shape rather than magnitude; and no source in this corpus publishes a fleet-wide error budget spent on deploys, which is an open number worth measuring in your own system.
The AWS and Envoy values are vendor documentation of their own defaults, reliable as defaults but not as recommendations. The Borg 80% figure is from 2013 data published in 2015; treat it as an order of magnitude, corroborated by nothing newer, because nobody else has published an equivalent rate.
Every source behind this page, graded. Filter by kind. The full
claim-by-claim ledger ships alongside this file as sources.md.
This dig ran in an environment whose network reaches github.com, gitlab.com and storage.googleapis.com directly; those sources were fetched in full. Sources on other hosts (marked “via search retrieval” in the ledger) were read through search-tool retrieval only, and their material is used as attributed reporting rather than verbatim quotation wherever wording could not be confirmed. No conference talk could be fetched at all, so the talk tier is empty: an access limitation of this run, not evidence that talks do not exist. KubeCon and SREcon programmes list several on this topic.
HAProxy server-slot exhaustion made the routing state progressively stale all day; the 48-minute outage fired when the evening scale-down removed instances the state no longer tracked correctly.
A chart port-rename reconfigured the GCP load balancer in place; 78 minutes of degraded service across five windows. Corrective action: build graceful draining at cluster granularity.
An untested ASG cycler script replaced instances without safe draining during a secrets migration; combined with a misconfigured endpoint, the site returned 502 for everything.
The design record that turned the community's sleep workaround into an API field, motivated by images with no sleep binary and by the endpoint-propagation race the sleep exists to lose deliberately.
Names the traffic-loss window on rolling updates and its two governing factors, local endpoint count and LB probe interval; kube-proxy learns to use terminating pods rather than drop connections.
Before this, terminating endpoints vanished from the API entirely; the
serving and terminating conditions make the lame-duck state
expressible so consumers can drain instead of guess.
L4 balancers cannot move load off idle HTTP/2 connections, so servers bound connection age and drain with GOAWAY, letting outstanding RPCs complete inside a grace window.
The merged implementation whose description states its purpose exactly: patch the race between when a pod is killed and when the external load balancer notices, by routing to terminating pods as a last resort.
The recorded, accepted, frozen ask to sequence SIGTERM behind LB deregistration and to extend readiness gates to deletion. Five years of activity around it produced softeners, never the barrier.
The cleanest public measurement of cloud-LB endpoint programming lag: 15–60 seconds of 502s per rollout with only one pod replaced, closed as not planned.
With a mesh sidecar, the app can die while the proxy still accepts for it; connection refused before the proxy even saw SIGTERM. The thread is a catalogue of ordering annotations.
Measured 400/502/504 mix at 500 QPS during a rolling update; no single knob fixed it, and the ecosystem's answer became webhook-injected pod readiness gates for the registration side.
The recorded design argument about socket handover when shared-memory formats change; without it, “the only way to do this would be something like a prolonged cluster drain for upgrades”. Open since 2018.
The handover contract stated as four constraints: no old code after success, an init grace for the new process, crashing during init is fine, one upgrade at a time.
A broker holds the LISTEN sockets and passes them over a UNIX domain socket, so HAProxy reloads reuse the same socket and the port never closes; identical binds get the identical socket.
An admission webhook intercepts pod deletion and delays it until deregistration has happened, because the sleep needs a binary the image may lack, needs chart patches everywhere, “and it is ugly”.
Client-side balancing lagged rollouts on its DNS refresh cycle exactly as an external LB lags on probes; 60s preStop and 180s grace did not save it. Closed as not planned.
Section 2.3: tasks can request SIGTERM notice before SIGKILL to finish requests and decline new ones, and in practice the notice arrives about 80% of the time.
The balancer fleet itself is upgraded by rolling restart with per-machine draining, an operation that can run over an hour; consistent hashing plus connection tracking keeps connections alive while the balancer set churns.
The counter-position: if software must recover from crashes anyway, crashing should be the only stop mechanism and recovery the only start. Right about state; the routing-plane half of removal is outside its frame.
The reference description of serving-but-leaving: the task listens and can serve while explicitly asking clients to stop; the state reaches clients in one or two round trips and exists to make clean shutdown simple.
The platform documents that endpoint removal is evaluated at the same time as shutdown begins, that deletes default to 30 seconds of grace, and that ordering is not guaranteed.
Protocol-aware drain: fail health checks, discourage traffic across the window
(600 s default, gradual ramp), Connection: close for HTTP/1, GOAWAY for
HTTP/2, old process killed at 900 s.
Deregistering targets enter draining; the delay defaults to 300 seconds (0–3600), and completes early only when no in-flight requests or active connections remain.
The motivating outage in the docs: a rollout where the target group holds only Initial or Draining targets. A mutating webhook injects a readiness condition tied to target health; deletion-side coverage is explicitly absent.
At GitHub's scale, the standard reload trick, drop SYNs briefly and let clients retry, caught a customer-impacting number of connections every time; multibinder's socket inheritance removed the window instead of shrinking it.
The pre-handover state of the art: delay SYN packets in a Linux qdisc during the reload so the new process answers them on release; an ingenious buffer where GitHub later built a bypass.
Reload-based config changes meant every HAProxy reload spawned a new process while the old one lingered for many hours draining websockets; a fleet of half-retired processes was one of the stated reasons to move to Envoy's dynamic config and built-in drain.
The community's canonical walkthrough of the race: endpoint propagation is eventually consistent across kube-proxies, ingresses and cloud LBs, unsequenced against SIGTERM, so delay the shutdown until routing catches up.
An operator's independent statement of the mechanism: no orchestration exists between the TERM signal and endpoint-list removal, the two happen in any order, and the preStop wait exists to absorb that.
Six rungs from an evening's toy to a measured production posture. The line from toy to real crosses at rung four.
A minimal HTTP server behind any load generator. Kill it with SIGKILL mid-load and count failed requests at the client. This number is your baseline, and every later rung is judged by how much of it survives.
Done when: you can state failed requests per kill, from client-side measurement. Teaches: errors are measured at the caller; the server's own logs cannot see a dropped connection.
Add a SIGTERM handler: stop accepting, finish in-flight, exit. Repeat the kill. In-flight failures disappear; connection-refused errors remain, because the load generator, like a load balancer, keeps dialing an address nobody told it to stop using.
Done when: failures are all connection-level, none mid-request. Teaches: the app can only fix the in-flight half; routing is someone else's state.
Add a reverse proxy with active health checks. On SIGTERM, fail the health endpoint first, keep serving, exit after the proxy stops sending. Vary probe interval and unhealthy threshold and plot the error window against them.
Done when: the measured window tracks probe interval times threshold, as KEP-1669 predicts. Teaches: polling drains are computable; you can size before you test.
Deploy on Kubernetes behind an ingress, run the load generator, and roll the deployment with and without a preStop sleep. Then replace the sleep with a webhook-based hold (pod-graceful-drain or equivalent) and compare error rates and rollout duration.
Done when: you have an errors-per-rollout and minutes-per-rollout table for all three configurations. Teaches: the trade the whole industry is making, on your own numbers.
Add a websocket or gRPC stream client that holds its connection. Roll again and watch drains stall or connections drop at the deadline. Fix it with server-initiated rotation: bounded connection age plus GOAWAY, and jitter the client reconnects.
Done when: a rollout completes inside its budget with zero mid-stream drops and no reconnect spike. Teaches: persistent connections turn drain into a protocol feature, not a timer.
Borg's number says one in five terminations gets no graceful notice. Make your rollout tooling SIGKILL a random 20% of instances instead of draining them, continuously, and verify the client-side error budget and any state invariants still hold.
Done when: the fleet passes its error budget with the 20% ungraceful rate left on permanently. Teaches: the drain is an optimisation; crash-safety is the contract, and this rung is the regression test for it.
The queries that found this material, grouped by what they surface. The vocabulary is the value: lame duck, deregistration delay, terminating endpoints, preStop.
"graceful shutdown" 502 rolling deploy preStop sleep "we"postmortem "rolling deploy" OR "rolling restart" 502 "we" -tutorial"terminating" pods "still receive traffic" kubernetesKEP sig-network terminating endpoints site:github.comrepo:kubernetes/kubernetes is:issue "SIGTERM" endpoint removal sequence"lame duck" "explicitly asking clients to stop sending requests"site:gitlab.com gl-infra production "incident review" drain OR deployment 5xxrepo:kubernetes/ingress-gce is:issue 502 rolling restartstatus page incident "connection draining" root cause"zero downtime" haproxy reload SO_REUSEPORT OR qdisc "we"envoy hot restart drain-time-s parent-shutdown-time-sgrpc GOAWAY "max connection age" rollout UNAVAILABLE