Three designs need distributed locks: a nightly report, a per-customer state machine, and a global config reload. For each, is a lock the right answer?

The nightly report — a lock is acceptable

Purpose: efficiency. Two instances generating the same report wastes compute and possibly sends two emails, but nothing is corrupted.

A lease-based lock is fine here, with two additions. Make the output idempotent — write to a deterministic location so a second run overwrites rather than duplicating — and make the notification idempotent on a deterministic key so the customer receives one email.

With those, the lock is an optimisation and correctness does not depend on it. That is the right relationship.

The per-customer state machine — a lock is the wrong answer

Purpose looks like correctness: two processes advancing the same customer's state concurrently would corrupt it. But a lock is not how to get it.

Partition instead. Route all events for a customer to the same consumer — Kafka partitioning by customer ID, consistent hashing, or a sharded worker assignment. Then each customer has exactly one owner at any time and mutual exclusion is a property of the routing rather than something you enforce per operation.

This removes the lock entirely, removes the lease-expiry problem, removes the coordination service from the request path, and is faster. Where a rebalance can briefly overlap ownership, an optimistic version check on the state write closes the gap.

The global config reload — a lock is the wrong tool

What is actually wanted is that a config change is applied consistently, not that one node applies it exclusively.

The right shapes: a watch on a config store where every node is notified and applies the change (etcd, Consul, or a config service), with a version number so nodes can confirm convergence; or a staged rollout so the change reaches instances progressively and a bad config is caught before it is universal — which is the more important property, given that global config pushes are a leading cause of large outages.

A lock here would serialise the reload without making it safer.

The general rule

Reach for distributed mutual exclusion only after ruling out: partitioned ownership, idempotency, and optimistic concurrency. In practice one of those three covers most cases, and each is cheaper and more robust than a lock.

When a lock genuinely is needed, name whether it is for efficiency or correctness — and if correctness, insist on fencing tokens enforced by the resource, or accept honestly that you have reduced probability rather than eliminated risk.

What a strong answer adds

Noting that the middle case is the most instructive: it looks like the strongest argument for a lock and is actually the clearest case for designing the need away. Most requirements for distributed mutual exclusion are requirements for single ownership, and single ownership is a routing decision.