Your service registry becomes unavailable. Every service is healthy. What happens, and what should happen?

What usually happens

A total outage, despite every service being fine — because resolution is on the request path and a failed resolution fails the request.

This is the textbook case of a control plane failure becoming a data plane outage. Discovery is a control-plane function: it decides where traffic goes. Serving a request with an address you already know is a data-plane function. If the second requires the first to be healthy, you have coupled them.

What should happen: nothing, for a while

Cache the last known good instance list and keep using it. A stale list is dramatically better than no list. Most instances in it are still valid; the ones that are not will fail individual requests, which retries and circuit breakers already handle.

This is static stability: the data plane continues on existing state when the control plane is unavailable.

The specifics worth designing:

Cache with a long fallback TTL — resolution results kept for minutes or hours, refreshed opportunistically, and used indefinitely if refresh fails rather than expiring into nothing.

Never fail closed on a resolution error. Log it, alert on it, and serve from cache.

Degrade gracefully as the list ages. If instances in the cached list start failing, remove them locally — the caller's own health observation is a valid substitute for the registry's.

Alert on staleness, so a registry outage is visible even though it is not causing user impact. Invisible degradation becomes a surprise when something else fails at the same time.

The related failures to check

Does the registry depend on itself? A registry whose clients discover it through the registry has a bootstrapping problem. It should be reachable at a stable address — DNS, a load balancer, or configuration.

Do health checks flap during a registry outage? If the registry is also the health-check aggregator, its outage can cause mass deregistration when it recovers — a thundering herd of re-registration, and a period where the list is empty rather than stale. Recovery should be gradual.

What is the deregistration delay? That interval — check period times threshold, plus propagation — is dead time in every caller's stream, and it is often tens of seconds when people assume it is one.

What a strong answer adds

Generalising the question: what does this system need to be working, right now, to serve a request it already knows how to serve? Every item on that list is an availability dependency, and discovery, configuration, feature flags, secrets and authorisation all commonly appear on it when they need not.