The router finds out last: taking a server out of service without dropping requests

Reconstructs the five-step removal sequence that Google, Kubernetes, AWS, Envoy and gRPC all converge on, and shows that implementations differ on exactly one axis: whether the drain closes the loop by subscription, half-closes it by polling, or runs open-loop on a timer. A reader leaves able to classify their own drain design, size its wait from measured propagation instead of folklore, and test the crash path that one in five removals takes anyway.

What you get out of it

• Removal is a three-party act with no shared transaction: the kill signal arrives in microseconds while the membership update replicates in seconds to minutes, and no acknowledgement flows back; every deploy-time 502 in the corpus is that gap observed from a different angle.
• Published drain designs differ on one axis, how the waiter learns it is safe: Google closes the loop by broadcast (1-2 RTT), health-check systems poll (probe interval times threshold, computable in advance per KEP-1669), and the industry default is an open-loop timer (preStop sleep, 300 s ALB deregistration delay, Envoy's 600 s drain).
• No fetched incident blames the drain sequence executing as designed; the outages come from the membership state being wrong (Slack 2020, GitLab 2022) or from tooling bypassing the drain contract entirely (an untested ASG cycler), so the risk lives in the state and the callers, not the mechanism.
• Persistent connections change the problem from a timer into a protocol feature: servers must rotate connections (GOAWAY plus bounded connection age, gRFC A9) because a connection that outlives the drain window pins its client to the dying server, as Slack's hours-long HAProxy zombie processes showed.
• Crash-safety is the floor, not an alternative: Borg delivers the SIGTERM notice about 80% of the time, so roughly one termination in five is ungraceful even at Google, and a fleet should be regression-tested at that rate.

Scope

Why this, now. KEP-3960 went stable in v1.34 (2025), making the community's decade-old preStop sleep workaround a first-class Kubernetes API field, which is the clearest statement yet that the platform's answer to the termination race is a timer, not a barrier.

What it does not cover. Failure detection (deciding an unresponsive peer is dead), stateful failover and session migration, batch-worker draining, and conference talks, which were unreachable from this run's network and are absent from the evidence wall for that stated reason.