Every deployment produces a small spike of 502s that the team has learned to ignore. Fix it.
Show the full answer Hide the answer
The cause
In-flight requests are being severed because the instance terminates before the load balancer has stopped sending to it, or before existing requests complete.
Two halves must cooperate, and the application half is almost always the one missing.
The correct shutdown sequence
On receiving SIGTERM, the application must:
- Fail its readiness check immediately. This is what tells the load balancer or platform to stop sending new requests. It must happen first.
- Keep serving in-flight requests — do not close listeners, do not exit.
- Wait for the deregistration delay to elapse, so the load balancer has observed the readiness failure and stopped routing.
- Then drain: finish outstanding work, close connections, flush buffers.
- Then exit.
The common bug is exiting immediately on SIGTERM. The platform is politely waiting for a process
that has already gone, and every request in flight at that moment fails.
The settings that must line up
Deregistration delay must exceed the longest legitimate request, or long requests are cut anyway.
Termination grace period (the time the platform allows between SIGTERM and SIGKILL) must exceed
the deregistration delay plus the drain time, or the process is killed mid-drain — which produces
exactly the symptom the drain was meant to remove.
Both delay every deployment and scale-in by their duration, so they should be as short as the longest request permits and no shorter.
The case that is usually missed
Long-lived connections. WebSocket and gRPC streams do not complete on their own, so a drain timeout never elapses naturally. These need an application-level signal telling clients to reconnect, and clients must reconnect with jittered backoff — otherwise every client of that instance reconnects simultaneously and the drain produces a thundering herd on the remaining instances.
What a strong answer adds
Pointing out that "the team has learned to ignore it" is the real finding. A recurring, explained-away error spike trains people to discount the error graph, which is the same graph that must be trusted during a genuine incident. Removing known-benign noise is worth doing for that reason alone.