Your cluster fails over spuriously under load, but a real leader failure takes 45 seconds to detect. How do you resolve the tension?

The tension

Detection time is set by the heartbeat interval times the failure threshold. Shorten it and you detect real failures faster and mistake slow-but-alive leaders for dead ones. Lengthen it and you get the reverse. There is no setting that is good at both, because a timeout genuinely cannot distinguish "crashed" from "slow".

How to resolve it

1. Fix the load-induced slowness first. Spurious failover under load usually means the leader is missing heartbeats because it is saturated — heartbeats queued behind application work, garbage collection pauses, or a shared thread pool. Sending heartbeats on a dedicated, high-priority path that does not share resources with request handling removes most spurious failovers without changing any timeout. This is the highest-value fix and it is frequently overlooked.

2. Then set the timeout from the observed distribution. Measure heartbeat delivery latency, including the tail under peak load and during GC pauses. Set the threshold comfortably above the p99.9 of that distribution — not above the average, which is what produces the spurious failovers.

3. Randomise the election timeout so that when an election does occur, split votes do not repeat and extend the outage.

4. Make the asymmetry explicit. Failing over unnecessarily costs an election, a leadership change, possible client disruption and — worst — a window in which two nodes believe they are leader. Failing over slowly costs 45 seconds of write unavailability. Un-failing-over is far more expensive than failing over slowly, so when in doubt, bias towards longer.

5. Make the 45 seconds not matter as much. Reduce the cost of detection latency rather than only the latency itself: queue writes client-side during a leadership change, serve reads from followers throughout, and ensure the new leader warms quickly.

The safety net that makes tuning less critical

Fencing tokens. With them, a deposed leader that has not yet realised it is deposed cannot do damage — its writes are rejected by the resource. That decouples correctness from timing, and it means the timeout can be tuned for availability rather than being the only thing preventing corruption.

What a strong answer adds

Noting the difference between detection and promotion. Detection can be fast and cheap; promotion can require a stronger signal — a quorum agreeing, or a human confirming for cross-region changes. Separating the two lets you notice quickly without acting rashly.