Every deployment produces a 30-second burst of connection errors, even though the new instances are healthy. The registry shows correct data. What is the most likely cause?
Show the full answer Hide the answer
What is being tested
Whether you understand that discovery data is cached somewhere you did not write, and that shutdown is a protocol rather than a signal.
The mechanism
Callers do not query the registry per request — that would be absurd. They cache the instance list, or their DNS resolver caches the record, or their runtime caches the resolution for the process lifetime. That cache has a lifetime measured in tens of seconds.
When an old instance receives its termination signal and immediately stops accepting connections, every caller holding a stale list keeps sending to it until their cache expires. Those requests fail. The registry is correct; the callers have not asked it recently.
The correct shutdown protocol
- Deregister first — remove yourself from the registry or fail the readiness probe.
- Keep serving. This is the counterintuitive step. Continue accepting and completing requests for at least as long as the longest plausible caller cache — typically the readiness probe interval plus the DNS TTL plus a margin.
- Stop accepting new connections, drain in-flight requests up to a deadline.
- Exit.
Most platforms give you a preStop hook or equivalent precisely for step 2. Teams routinely skip
it because it looks like a pointless sleep, and then accept a burst of deployment errors as normal.
It is not normal.
The related traps worth naming
Runtime DNS caching. Some language runtimes cache DNS resolution for the life of the process, ignoring TTL entirely. Traffic then goes to a dead address indefinitely, not for 30 seconds. This is worth checking explicitly for every language in the estate.
Health checks that test dependencies. If a liveness probe calls a downstream service, a downstream slowdown makes healthy instances report unhealthy. The platform removes them, capacity drops, remaining instances get more load and also fail their probes. A dependency's slowdown becomes your total outage. Liveness should test the process; readiness may test dependencies, and they must be different endpoints.
A strongly consistent registry. Making the registry CP means it becomes a single point of failure for everything. Netflix's design deliberately prefers stale-but-usable instance data over no data: if the registry is down, callers keep using their last known list. A stale list means some calls fail fast and are retried. An empty list means total outage.
Why the other options are wrong
Failing health checks would show unhealthy instances in the registry, and the question says the registry is correct. A load balancer misconfiguration would not resolve itself after 30 seconds. A small connection pool produces latency and saturation, not a burst of connection errors timed to deployments.