Lock Lease Expiry
The timeout on a distributed lock that prevents a crashed holder deadlocking the system — and the source of the pattern's hardest failure mode.
A distributed lock must expire, or a holder that crashes while holding it blocks everyone forever. But an expiring lock creates the opposite hazard: the holder may still be working when the lock is handed to someone else.
Choosing the duration is a trap either way. Too short and long operations lose their lock mid-work. Too long and a crash blocks progress for that duration. And the "correct" value depends on the maximum time the operation might take, which is precisely the thing you cannot bound in a system with garbage collection, network delay and slow disks.
The mitigations, none of which fully solve it: heartbeat renewal, where the holder extends the lease while working — which helps with slow operations and not with pauses. Fencing tokens, which make a stale holder's writes harmless. And making the guarded operation idempotent, which is the only approach that removes the requirement for the lock to be correct at all.
The design conclusion most teams eventually reach: prefer an architecture that does not need distributed mutual exclusion — partition the work so each item has one owner — over one that does.