A team uses a distributed lock in a key-value store to ensure only one worker processes a job. Occasionally two workers process the same job. Explain.
Show the full answer Hide the answer
What the interviewer is testing
Whether you know the fundamental flaw in lease-based distributed locking, which no amount of configuration fixes.
The mechanism
Worker A acquires the lock with a 30-second lease and starts work. A then pauses — a garbage collection pause, a scheduler preemption, a hypervisor migration, a network stall.
The lease expires. Worker B legitimately acquires the lock and begins processing.
A wakes up. From its perspective no time has passed and it still holds the lock, so it writes.
Two workers, mutual exclusion violated, and no component did anything wrong. Increasing the TTL does not fix it — it only changes how long a pause must be, and pauses of many seconds are entirely possible.
The fix: fencing tokens
The lock service issues a strictly increasing number with each grant. Every write carries the token, and the resource being protected rejects any token lower than the highest it has seen.
A wakes with token 33, B holds 34, the storage layer refuses A's write. Correctness now sits at the resource rather than depending on the client's belief about time.
The limitation to state
This only works if the protected resource can perform the check. Where it cannot — a filesystem, a legacy API, a third-party service — distributed locking cannot be made safe.
In those cases the honest answer is to restructure so that mutual exclusion is not required:
- Make the operation idempotent, so duplicate execution is harmless
- Partition ownership so only one worker can ever hold a given job by construction — assign jobs by consistent hash, or use a partitioned log where each partition has one consumer
- Use a conditional write on the target with an expected-state check, which is fencing by another name
What a strong answer adds
Preferring the partition-ownership approach generally. A design where the exclusive owner is determined structurally has no lock to expire and no failure mode to reason about, and it is usually available.
Common weak answers
Lengthening the lease. Adding a heartbeat to renew the lock, which does not help a paused process.