A nightly job occasionally runs twice, producing duplicate charges. The team proposes a distributed lock. What do you say?

The first response

A lock will reduce the frequency and will not eliminate it, and if the team believes otherwise they will stop looking for the real fix.

The reason is the standard one: a lock holder that pauses — garbage collection, a hypervisor stall, a slow disk — can have its lease expire while it is still working. A second holder starts. The first resumes, believing it holds the lock, and both charge.

So the question to ask is which of two things they are buying: efficiency (avoiding duplicate work, where a rare double execution is merely wasteful) or correctness (where a duplicate charge is unacceptable). This is clearly the second, and a lock alone never provides it.

What actually fixes it

Idempotency, at the level of the effect. Each charge carries a deterministic key — for example a hash of (customer, billing_period, invoice_id) — and the payment system rejects a duplicate key. Then it does not matter how many times the job runs: the charge happens once.

This is the fix that works regardless of scheduling, retries, redeployments, manual re-runs and operator error, and it is the only one that does.

Fencing tokens, if the resource can enforce them: the lock service issues a monotonically increasing token, and the downstream system rejects writes carrying a stale one. This makes a deposed holder harmless rather than merely unlikely.

Partitioned ownership as the structural alternative: rather than one global job needing mutual exclusion, partition customers across workers so each customer has exactly one owner and no coordination is required at all.

What the lock is still good for

Keeping it is reasonable — as an efficiency measure. It prevents most duplicate work, which saves resources and reduces load. It should just not be the thing correctness depends on.

What a strong answer adds

Asking why it runs twice in the first place. Usually the answer is that the scheduler retried after a timeout, or two instances both fired, or someone re-ran it manually after a partial failure. Each of those is worth fixing on its own, and the last one — a manual re-run after a partial failure — is the case that idempotency handles and a lock never will.