Distributed Systems 19 Sep 2026 26 min read

When two hold the lock: what the repositories admit about distributed mutual exclusion

How production systems try to guarantee that at most one process acts at a time, reconstructed from the lock implementations' own code comments, documentation, issues, merged and rejected pull requests, design records, advisories and public incident trackers.

Every mainstream distributed lock ships with a written admission that it cannot guarantee mutual exclusion, and this guide reads those admissions where they live: client-go's package comment, etcd's post-Jepsen documentation, Consul's session internals, Redis's own Redlock page, plus the incidents where the lease machinery itself produced two holders. An architect leaves with a decision tree (efficiency lock, fencing token, or self-fencing exit), the TTL numbers the industry actually runs, and a failure catalogue with a design rule per incident.

The finding that surprised me

The industry's real safety mechanism is not the lock but deliberate process death: a rejected Kubernetes pull request records the design intent in one line, 'without fencing we need to exit(1) as fast as possible', and Patroni escalates the same idea to kernel watchdog reboots.

What you get out of it

  • Every implementation in the corpus disclaims itself in its own repository: client-go 'does not guarantee that only one client is acting as a leader', etcd's docs say the lease mechanism 'doesn't guarantee mutual exclusion' and cannot protect external resources, Consul calls its safeguard 'not a bulletproof method', and Redis's Redlock page prints 'SAFETY VIOLATION!' for the failover race.
  • Fencing is one mechanism with many names, and etcd's documentation writes the genealogy down: Chubby's sequencer, Kleppmann's fencing token, etcd's revision, ZooKeeper's zxid, CockroachDB's liveness epoch, Kafka's producer epoch. It ships only where one project owns both the lock and the resource.
  • The lock service gets the same disease it treats: a stuck etcd leader kept mass-revoking leases after being deposed (issue 15247, fixed 2023), and a CockroachDB lease-expiry regression produced two simultaneous leaseholders and irrecoverably lost writes across two years of releases (advisory a131639, 2024).
  • On lease loss the recorded industry answer is death, not recovery: Kubernetes merged exit-on-loss for the scheduler in three days in 2019, the opposite proposal was NAKed in 2017, HBase documents shutdown-on-session-expiry as a design decision, and Patroni backs demotion with a kernel watchdog whose own residual hole is documented.
  • GitLab's production tracker shows the split-brain detector (count of read-write primaries) firing five times from 2022 to 2026 with at least three false positives from provisioning and monitoring changes, and no public account in the corpus records data corrupted through an unfenced TTL lease at the application tier, which argues for making double execution cheap because it will not be reliably detected.

Scope

Why this, now. Leader election and job-deduplication locks sit under most Kubernetes-era platforms, and the corpus shows the same zombie-holder failure recurring from 2016 to a 2024 production data-loss advisory, so the risk is current even though the argument is a decade old.

What it does not cover. Consensus-protocol internals, transaction isolation, and everything published only on blogs, in papers or as talks: the session's network reached code hosts only, so the Kleppmann and antirez posts, the Chubby paper, the Jepsen report and the Roblox, GitHub and Cloudflare outage write-ups appear solely through the repository artefacts that quote or implement them.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.