Progress-Based Liveness

A heartbeat answers "is the process running." That is the wrong question, because the most damaging failure in a leader-based or worker-based system is a process that is alive, responsive, and not making progress — a long garbage-collection pause, a saturated disk, a stalled downstream, a lock it cannot acquire.

Such a process keeps its lease, so no election is triggered, and it serves every request badly. The system is formally healthy and functionally down.

Why it matters

It is the failure mode that heartbeat-based detection is structurally incapable of seeing, and it is more common than outright crashes. Crashes are handled well by every framework; brownouts are handled by almost none.

Implementation patterns

• Health checks that assert forward motion: is the commit index advancing, is the queue draining, is the last-completed-work timestamp recent, is latency within bound.
• Have the process step down voluntarily when it fails its own progress check, rather than waiting to be evicted. Self-demotion is faster and safer than external detection, because the process has more information about its own state than any observer does.
• Separate liveness from readiness. A process that is alive but not progressing should stop receiving new work without being killed, so it can be inspected.
• Bound the lease and monitor clock drift, since lease safety depends on an assumption about clocks that must be verified rather than trusted.
• Pair with fencing tokens. Detection tells you to stop the old leader; fencing stops its already-issued writes from landing, and it is the half most often omitted.
• Alert on the absence of progress, not only on errors. A stalled component produces silence, and silence resembles success on most dashboards.

Industry example

Blockchain and distributed-ledger infrastructure such as Polygon's validator and RPC layers depends on this distinction directly: a node that is up but not keeping pace with the chain head is worse than a node that is down, because traffic keeps being routed to it and it returns stale answers confidently. The health signal that matters is block height relative to the network, not process liveness.

The same pattern applies to any consumer of a log — a stream processor whose lag is growing, a replica whose apply queue is falling behind, a worker pool whose oldest in-flight task keeps getting older.

Failure scenarios

• Heartbeat passes, work stops. The canonical case.
• A leader keeping its lease through a long pause, so no failover occurs.
• Load balancers keeping a brownout instance in rotation because it answers the health endpoint.
• Detection without fencing, so the demoted leader's in-flight writes still land.
• Progress checks measuring the wrong thing — request count rather than useful work completed — which a stuck process can satisfy by failing quickly.

Trade-offs

Progress checks are more likely to produce false positives than heartbeats: a legitimately idle system makes no progress, and a temporary downstream stall is not the leader's fault. Demoting on those causes unnecessary failovers, and failovers are not free — they pause work and can cascade.

The mitigations are hysteresis (require sustained absence of progress), distinguishing "no work available" from "work available and not progressing", and making the first response a readiness change rather than a demotion. A system that fails over too eagerly is its own outage, so the tuning matters as much as the mechanism.

Interview question

"Your leader is passing every health check and the cluster is not making progress. What checks would have caught this, what should the leader itself have done, and why is detection alone not sufficient?"