When two hold the lock: what the repositories admit about distributed mutual exclusion
How production systems try to guarantee that at most one process acts at a time, reconstructed from the lock implementations' own code comments, documentation, issues, merged and rejected pull requests, design records, advisories and public incident trackers.
Every mainstream distributed lock ships with a written admission that it cannot guarantee mutual exclusion, and this guide reads those admissions where they live: client-go's package comment, etcd's post-Jepsen documentation, Consul's session internals, Redis's own Redlock page, plus the incidents where the lease machinery itself produced two holders. An architect leaves with a decision tree (efficiency lock, fencing token, or self-fencing exit), the TTL numbers the industry actually runs, and a failure catalogue with a design rule per incident.
The industry's real safety mechanism is not the lock but deliberate process death: a rejected Kubernetes pull request records the design intent in one line, 'without fencing we need to exit(1) as fast as possible', and Patroni escalates the same idea to kernel watchdog reboots.
What you get out of it
- Every implementation in the corpus disclaims itself in its own repository: client-go 'does not guarantee that only one client is acting as a leader', etcd's docs say the lease mechanism 'doesn't guarantee mutual exclusion' and cannot protect external resources, Consul calls its safeguard 'not a bulletproof method', and Redis's Redlock page prints 'SAFETY VIOLATION!' for the failover race.
- Fencing is one mechanism with many names, and etcd's documentation writes the genealogy down: Chubby's sequencer, Kleppmann's fencing token, etcd's revision, ZooKeeper's zxid, CockroachDB's liveness epoch, Kafka's producer epoch. It ships only where one project owns both the lock and the resource.
- The lock service gets the same disease it treats: a stuck etcd leader kept mass-revoking leases after being deposed (issue 15247, fixed 2023), and a CockroachDB lease-expiry regression produced two simultaneous leaseholders and irrecoverably lost writes across two years of releases (advisory a131639, 2024).
- On lease loss the recorded industry answer is death, not recovery: Kubernetes merged exit-on-loss for the scheduler in three days in 2019, the opposite proposal was NAKed in 2017, HBase documents shutdown-on-session-expiry as a design decision, and Patroni backs demotion with a kernel watchdog whose own residual hole is documented.
- GitLab's production tracker shows the split-brain detector (count of read-write primaries) firing five times from 2022 to 2026 with at least three false positives from provisioning and monitoring changes, and no public account in the corpus records data corrupted through an unfenced TTL lease at the application tier, which argues for making double execution cheap because it will not be reliably detected.
Scope
Why this, now. Leader election and job-deduplication locks sit under most Kubernetes-era platforms, and the corpus shows the same zombie-holder failure recurring from 2016 to a 2024 production data-loss advisory, so the risk is current even though the argument is a decade old.
What it does not cover. Consensus-protocol internals, transaction isolation, and everything published only on blogs, in papers or as talks: the session's network reached code hosts only, so the Kleppmann and antirez posts, the Chubby paper, the Jepsen report and the Roblox, GitHub and Cloudflare outage write-ups appear solely through the repository artefacts that quote or implement them.
Other field guides
When the answer is wrong, not late: ten years of CockroachDB read from its own record
A decade of one distributed database company read from artefacts nobody edits for effect: technical advisories with affected version ranges, design r…
26 sources · 3 organisations · 8 postmortemsTime cannot go backwards, and other lies production believed
Two postmortems (Cloudflare 2017, Azure leap day 2012), three Linux kernel commits, the Go and Rust standard-library design records, Kubernetes leade…
23 sources · 14 organisations · 2 postmortemsA cache miss is a protocol, not an event
Reconstructs the miss protocol behind every production cache from Facebook, Slack and Wikimedia postmortems and the implementations at Google, Instag…
26 sources · 22 organisations · 3 postmortems