Distributed Locks in Practice

Definition

A mechanism ensuring that at most one process across a distributed system holds exclusive access to a resource at a time.

Why it matters

Because the naive implementation is unsafe in a way that testing will not reveal, and the failure is silent data corruption rather than an error.

The scenario: process A acquires a lock with a 30-second lease and begins work. A pauses — garbage collection, scheduler preemption, hypervisor migration, a network stall. The lease expires. Process B legitimately acquires the lock. A resumes, still believing it holds the lock, and writes.

Two writers. Mutual exclusion violated. No component did anything wrong, and increasing the lease duration does not fix it — it only changes how long a pause must be, and multi-second pauses are entirely normal.

Implementation patterns

Lease with fencing tokens. The lock service issues a strictly increasing number with each grant. Every write carries the token, and the protected resource rejects any token lower than the highest it has seen. A wakes with token 33, B holds 34, the storage layer refuses A's write. Correctness now lives at the resource rather than depending on the client's belief about time.

Consensus-backed locks using a system with a proper quorum, which handles the coordination correctly — and still requires fencing at the resource for the pause scenario.

Database-level locking where the work is confined to one database — SELECT FOR UPDATE, advisory locks, or a unique constraint. Simplest and safest when applicable, because the exclusion and the effect are in the same transactional boundary.

Failure scenarios

No fencing, which is the default and is unsafe as described.

The resource cannot check a token. A filesystem, a legacy API, a third-party service. Here distributed locking cannot be made safe by any configuration, and it is important to say so rather than tuning timeouts.

Lock service unavailability blocking all work, unless a degraded mode is defined.

Lock held across a long operation, so a crash blocks progress until the lease expires — and a long lease means long blocking while a short one means the pause scenario is more likely.

Industry example

The pattern is common enough that its unsafety has been publicly debated at length in the context of lock implementations built on distributed caches. The consensus of that debate is the useful takeaway: a lock built for efficiency (avoiding duplicated work, where a rare duplicate is merely wasteful) has different requirements from one built for correctness (where a duplicate corrupts data). Most implementations are adequate for the first and inadequate for the second, and teams routinely use them for the second.

Trade-offs

Locks make exclusive access explicit and introduce a coordination dependency, a failure mode, and a performance ceiling — the lock serialises access, so throughput on the protected resource is bounded by one holder.

The alternative is usually better: restructure so exclusion is not required. Make the operation idempotent so duplicates are harmless. Partition ownership so only one worker can ever hold a given item by construction — assign by consistent hash, or use a partitioned log where each partition has one consumer. Use a conditional write on the target, which is fencing by another name.

Interview question

A team uses a distributed lock to ensure one worker processes each job. Occasionally two workers process the same job. Explain.

The candidate must reach process pause versus lease expiry, and recognise that no timeout setting fixes it. Strong answers propose fencing tokens, then note the precondition — the resource must be able to check them — and offer partitioned ownership as the design that removes the need for a lock at all.