Lease
Time-bounded leadership or ownership that must be renewed before it expires, so a leader that becomes unreachable automatically relinquishes its role.
The mechanism that makes leadership recoverable without a human. A leader holds a lease for a fixed duration and renews it periodically; if it stops renewing — because it crashed, was partitioned, or paused — the lease expires and a new leader can be elected.
The subtlety that causes outages is that leases depend on clocks, and clocks are the least trustworthy thing in a distributed system. A leader that experiences a long garbage-collection pause may wake believing it still holds a lease that expired while it was stopped. It then acts as leader simultaneously with the newly elected one.
Two mitigations. Leaders should check the remaining lease time immediately before acting, not just periodically, and refuse to act if the margin is thin. And the downstream resource should enforce fencing tokens, so that even a leader that is wrong about holding the lease has its writes rejected.
The design rule that follows: never let a lease be the only thing preventing two leaders from writing. Time-based safety is a liveness optimisation, not a correctness guarantee.