A team proposes a distributed lock to stop two workers processing the same transaction. Workers sometimes crash or pause while holding the lock. What is the safer design, and when is a lock genuinely necessary?

Why the lock is fragile here

A lock held by a process that has stopped is a deadlock, so locks need leases. A lease that expires while the holder is merely paused — a long garbage-collection pause, a stalled disk write, a network partition — means two workers hold the lock simultaneously and neither knows it. The lock provided a feeling of exclusivity that it cannot actually deliver, which is more dangerous than no lock at all because the code downstream assumes it.

If a lock is genuinely required, it needs fencing tokens: a monotonically increasing number issued with the lease and checked by the resource being protected, so a write from a stale holder is rejected at the point of effect rather than at the point of intent. Without fencing, a lease-based lock is a performance optimisation, not a correctness mechanism — and it should be described that way in the design.

The safer design

Make the operation idempotent and let it run twice. For a financial transaction that means a natural idempotency key — the transaction ID — and a durable record checked and written in the same transaction as the effect. Two workers both process it; one wins the insert, the other finds the record and returns the same result. No coordination service, no lease tuning, no split-brain, and one fewer runtime dependency.

Coordination is a cost; idempotency is a property. Given a choice, buy the property.

When a lock is still the right answer

• When the effect is genuinely external and cannot be deduplicated — sending a physical letter, calling an API with no idempotency support, actuating a device. Here you are choosing between duplicate effects and coordination risk, and coordination may be the lesser evil.
• When duplicate work is prohibitively expensive even though it is safe — an eight-hour job you do not want to run twice. This is an efficiency argument, and it should be made explicitly as one, because it changes what happens when the lock fails: you accept the duplicate rather than blocking.
• When exclusivity is the business rule itself, such as single ownership of a shard.

The framing to use in review

"What breaks if this runs twice?" If the answer is "nothing, it is idempotent", the lock is unnecessary complexity. If the answer is "a customer is charged twice", the fix is the idempotency key, not the lock — the lock only narrows the window. A lock as the sole protection against a duplicate financial effect is a design that has decided a rare failure is acceptable without saying so.