A liveness probe checks the database. The database slows down. Describe what happens.

What is being tested

The classic self-inflicted amplification, and whether you know the correct separation of probe responsibilities.

The sequence

1. The database slows. Application instances are degraded but functioning — some requests succeed, some are slow.
2. The liveness probe calls the database. It exceeds its timeout. The instance reports not alive.
3. The platform does what liveness means: it kills and restarts the instance.
4. Every instance is affected identically, because they all share the same database. All of them fail the probe. All of them are restarted.
5. Restarted instances have cold caches and empty connection pools. They immediately open new connections to a database that is already struggling, adding connection setup load to a saturated system.
6. They fail the probe again — the database has not recovered and is now worse. Restart loop.
7. A degraded dependency has become a total outage, executed by your own orchestrator on your behalf.

The system had a partial problem and the health check converted it into a complete one.

The correct separation

Liveness answers: is this process broken beyond recovery and in need of a restart? It tests the process only — is the event loop responsive, is the thread pool deadlocked. Nothing external. Restarting is a violent action and should be reserved for states that only a restart fixes.

Readiness answers: should this instance receive traffic right now? This may consider dependencies, because failing readiness removes the instance from load balancing without killing it, which is recoverable. Even so, use it carefully: if every instance fails readiness simultaneously, the service is down. Some platforms have a "if all are unhealthy, route to all anyway" behaviour, which is worth knowing exists.

Startup answers: has initialisation finished? Without a separate startup probe, a slow-starting instance is killed repeatedly in a loop.

The other health-check rules

• Probes must be cheap. They run every few seconds on every instance; a probe that queries the database generates significant load and can cause the problem it reports.
• Deregister before shutdown. Fail readiness, keep serving for at least the longest plausible caller cache, then stop accepting and drain. Skipping this is why deployments produce a burst of errors.
• Probe timeout shorter than the probe interval, or probes queue up.

The general principle

Automated remediation must be more careful than the failure it responds to. Any mechanism that takes destructive action based on a signal that can be triggered by an external dependency will eventually amplify an incident. The same reasoning applies to autoscaling that reacts to a metric a downstream can move, and to circuit breakers whose recovery probe hammers a struggling service.