A monolith is moved to Kubernetes and the team discovers pod restarts drop in-flight requests and kill long-running jobs. What is actually happening, and what must be configured?

What is actually happening

Pod termination and endpoint removal are concurrent, not sequential. When a pod is deleted:

• The API server marks it terminating and notifies the kubelet, which sends SIGTERM.
• Independently, the endpoints controller removes it from the Service, and that removal must then propagate to every kube-proxy or ingress controller.

Nothing orders these. The application often receives SIGTERM and exits before the load balancers have learned to stop sending it traffic — so requests arrive at a socket that is closing. The dropped requests are not a bug in the application; they are a race in the platform that the application must absorb.

What must be configured

• A preStop hook that sleeps for longer than endpoint propagation typically takes (a few seconds). This is the counterintuitive but standard fix: the pod keeps serving during the sleep while the removal propagates, and only then does SIGTERM arrive.
• Handle SIGTERM by draining, not by exiting: stop accepting new connections, finish in-flight requests, close upstream connections cleanly, then exit.
• terminationGracePeriodSeconds longer than the preStop sleep plus the longest in-flight request. If it is shorter, the kubelet sends SIGKILL mid-request and the careful draining is irrelevant.
• A readiness probe that fails immediately on shutdown initiation, giving a second signal to stop routing.
• Keep-alive handling: clients holding persistent connections will not re-resolve, so the server must send Connection: close or a GOAWAY frame during drain, or the client keeps using a socket to a dying pod. This is the most frequently missed part.

Long-running jobs are a different problem

Graceful shutdown does not help a job that runs for two hours; you cannot drain it in thirty seconds. The options:

• Checkpoint and resume. The job records progress and a restarted instance continues from the checkpoint. This is the only approach that also survives node failure, which will happen regardless of how the deploy behaves.
• Make work items small and idempotent, so the unit of loss is one item and re-processing is safe. Long jobs become many short ones, and the platform's assumptions become true.
• Separate the deployment lifecycle: run long jobs as Kubernetes Jobs rather than in the Deployment's pods, so a rolling update of the service does not touch them. Mixing request serving and long batch work in one pod guarantees this problem.
• PodDisruptionBudget to limit voluntary disruptions, which constrains drains and upgrades — but does not protect against node failure.

The framing to take away

Kubernetes assumes workloads are interruptible. Everything about its scheduling, autoscaling, spot-instance support and upgrade model rests on that assumption. Work that cannot tolerate interruption is fighting the platform, and the sustainable fix is to make the work interruptible rather than to make the platform gentle — because node failures, preemptions and upgrades will interrupt it anyway.