Lease

A plain distributed lock has a fatal liveness problem: if the holder crashes, the lock is held forever and the protected work never runs again. A lease fixes liveness by attaching an expiry — the holder must renew, and if it stops renewing the lease lapses and someone else may acquire it.

In doing so it trades a liveness problem for a safety problem, and understanding that trade is the whole point of the concept.

The safety problem it creates

A lease can expire while the holder still believes it holds it. The holder is not malicious and has not crashed — it was simply not running:

1. Worker A acquires a 30-second lease and begins work.
2. Worker A stops for 40 seconds: a garbage-collection pause, a suspended virtual machine, a blocked disk, a network blackout.
3. The lease expires. Worker B legitimately acquires it and begins the same work.
4. Worker A resumes, unaware that time passed, completes its work and writes its result.

Both workers followed the protocol exactly. No lease implementation can prevent this, because the lease service cannot reach into a paused process and stop it. Tightening the timeout makes it more likely, not less.

Why it matters

This is the same structural problem as split-brain in leader election, and it has the same solution. Recognising that a lease provides mutual exclusion among well-behaved processes — a much weaker guarantee than mutual exclusion — is what separates a design that is safe from one that is merely usually safe.

Implementation patterns

• Fencing tokens. Each grant carries a monotonically increasing number, passed to every downstream effect. The resource being protected records the highest token it has seen and rejects lower ones. This is what makes the construction safe; the lease alone never was.
• Renewal with self-abort. The holder renews periodically and verifies it still holds the lease; if renewal fails it abandons its work rather than continuing optimistically. Cheap, and it closes the common case.
• Lease duration sized against realistic pause times, not against the happy path — long enough to survive an ordinary GC pause, short enough that a crash does not stall the system unacceptably.
• Idempotent protected work, so a duplicate execution costs compute rather than correctness.
• Clock assumptions made explicit and monitored, since lease safety depends on bounded drift.

Industry example

Consider a media platform where expensive renders are protected by a lease so two workers do not process the same job. Workers occasionally crash mid-render, and occasionally two workers produce the same export anyway.

The crash is the liveness case the lease was introduced to fix. The duplicate is the safety case the lease introduced. The correct design keeps the lease as an optimisation that avoids wasted compute, and places correctness elsewhere: fencing tokens checked at the storage write, a deterministic artefact location so two renders produce the same object, and a conditional completion record.

Often the better tool is a queue with visibility timeouts rather than a lock at all — same exclusivity in practice, fewer moving parts, and it names the real requirement honestly: at-least-once execution with idempotent effects.

Failure scenarios

• Fencing implemented in the lock service but not in the data path, so the check happens where the damage does not.
• Lease renewal on the same thread as the work, so a long operation blocks its own renewal and loses the lease it is actively using.
• Treating the lease as a correctness guarantee, which produces a system that is correct only while nothing pauses.
• Timeouts tuned to the happy path, causing constant spurious expiry under ordinary load.

Trade-offs

Leases buy liveness — the system recovers from a dead holder without operator intervention — at the cost of a window in which two holders can believe they are exclusive. Shorter leases recover faster and widen that window; longer leases narrow it and stall longer after a crash. There is no setting that removes the trade, which is why the durable answer is fencing plus idempotency rather than tuning.

Interview question

"Your worker holds a 30-second lease and experiences a 45-second GC pause. Another worker picks up the job. What goes wrong, why can no lease timeout fix it, and what would you add so the system is safe regardless?"