A media platform uses a distributed lock to stop two workers rendering the same expensive export. Workers sometimes crash while holding the lock, and occasionally two workers process the same job anyway. Diagnose both problems and redesign.

Problem one: the crash

A lock held by a crashed process is held forever, and the job never runs. The naive fix — no expiry — converts a crash into a permanent stall.

The standard fix is a lease: the lock expires unless renewed. But this creates problem two.

Problem two: why two workers process the same job

With leases, this sequence is not merely possible but inevitable at scale:

1. Worker A acquires a 30-second lease and starts rendering.
2. Worker A experiences a long stop — garbage collection pause, VM suspension, disk stall, network blackout — for 40 seconds.
3. The lease expires. Worker B acquires it legitimately and starts rendering.
4. Worker A resumes. It has no idea time passed. It believes it holds the lock, finishes the render, and writes the output.

Both workers did exactly what the protocol allowed. No lock service can prevent this, because the lock cannot reach into a paused process and stop it. This is the same structural problem as split-brain in leader election, and it has the same answer.

The redesign

1. Fencing tokens. Each lease grant carries a monotonically increasing token. The worker passes it to every downstream effect — the storage write, the database update, the notification. Storage records the highest token seen and rejects lower ones. Worker A's late write is refused because B's token is higher. This is what makes the lock safe; the lock alone never was.

2. Idempotent output. Write the artefact to a deterministic, content-or-job-addressed location so that two renders of the same job produce the same object, and completion is recorded conditionally ("set status to complete if it is not already complete, and record which token did it").

3. Stop treating the lock as the correctness mechanism. The most valuable reframing here: the lock is an optimisation that avoids duplicated compute. Correctness comes from idempotency and fencing. A system that is correct only while the lock behaves is a system that is incorrect, because the lock will eventually not behave.

4. Heartbeat with self-abort. The worker renews the lease periodically and checks it still holds it. If renewal fails, it aborts its own work rather than continuing optimistically — cheap, and it closes the common case even though it cannot close the paused-process case.

The design worth considering instead

For a queue of render jobs, a distributed lock is often the wrong tool entirely. A queue with visibility timeouts and at-least-once delivery gives the same exclusivity property with fewer moving parts, and it makes the real requirement explicit: at-least-once execution with idempotent effects. The lock was an attempt to buy exactly-once, which is not for sale.

The takeaway

Distributed locks do not provide mutual exclusion in the presence of process pauses — they provide it in the presence of well-behaved processes, which is a different and much weaker guarantee. Design so that a lock failure costs money (duplicate work) rather than correctness (duplicate effects).