Every mainstream distributed lock ships with a written admission that it cannot guarantee
mutual exclusion. This guide reads those admissions where they live, in the repositories:
code comments, rejected pull requests, design records and incident trackers from
Kubernetes, etcd, CockroachDB, GitLab, HashiCorp, Redis and Apache. What comes out is a
usable rule set for when a lock is enough, when you need fencing, and why the industry's
real safety mechanism is exit(1).
Several processes, any of which can stall, crash or lose its network at any moment, must agree that at most one of them performs a given action at a time. That is the whole problem, and it is harder than it reads.
The surprising thing this dig turned up is not that distributed locks can fail. It is that the people who build them say so, in writing, in the artefact itself, and have for a decade. The package comment of the leader election library inside every Kubernetes control plane opens with: "This implementation does not guarantee that only one client is acting as a leader (a.k.a. fencing)" [1]. etcd's documentation states "Actually, the lease mechanism itself doesn't guarantee mutual exclusion" [8]. Consul's session documentation calls its locks "advisory mechanisms for mutual exclusion" and describes its main safeguard as "not a bulletproof method" [13]. Redis's own Redlock page walks through a replica failover and prints, in bold, "SAFETY VIOLATION!" [15]. These are not blog critiques. They are the vendors' and maintainers' own words, sitting next to the code people depend on.
So the practical question for an architect is not "which lock is safe". None of them is, alone, and every implementation says so. The question is what production systems actually do about it, and the repositories record three answers: push a monotonic token check into the resource being protected (fencing); make the deposed holder die before it can do harm (self-fencing, up to and including kernel watchdog reboots); or accept occasional double execution and design the work to tolerate it. Sections 2 to 4 trace who does which, and what happened when the machinery itself broke.
Scope, and an honest constraint. This guide covers lease-based mutual exclusion and leader election between processes in a datacenter: lock services, leases, fencing, and the failure modes of each. It does not cover consensus-protocol internals, database transaction isolation, or application idempotency design except where teams use it in place of a lock. One constraint shaped the corpus: this session's network reached code hosts only, so the famous blog-side of this argument (Martin Kleppmann's 2016 fencing post, Salvatore Sanfilippo's reply, the Jepsen etcd report, the Chubby paper, and the Roblox, GitHub and Cloudflare outage write-ups) is cited here only through the repository artefacts that quote, link or implement it. Those artefacts turn out to be the more durable record: the debate's conclusions were committed to the repos, with attribution, by the projects themselves [9].
Across ZooKeeper, etcd, Consul, Kubernetes, Patroni, Redis, DynamoDB and CockroachDB, the same five components recur. Two of them are routinely missing, and those two are where the incidents live.
The store is quorum-replicated in every correctness-oriented system: ZooKeeper's sequence nodes [14], etcd's revisioned keyspace, Consul's sessions, and the Kubernetes Lease object, which is etcd wearing an API. Redis is the deliberate outlier: its own documentation shows that a single instance with asynchronous replication loses the safety property the moment a failover promotes a replica that never saw the lock key, which is the race Redlock's five independent masters exist to close [15]. The store can always order its own records. Ordering yours is the open part.
The lease is a TTL measured in physical time, and every project that thought about it carefully says the same uncomfortable thing: the clock that expires the lease is not the clock of the process holding it. etcd: "Both of the server and client measures passing of time with their own clocks. It allows a situation that the server revokes the lease but the client still claims it owns the lease" [8]. Consul adds that TTL is only a lower bound; expiry may arrive late, and clients "should be aware of clock skew issues" [13]. The DynamoDB lock client goes furthest and stores no absolute time at all: an observer starts its own timer and expires a lock only after watching the record sit unchanged for a full lease duration, so "even if two different machines disagree about what time it is, they will still avoid clobbering each other's locks" [16]. Note what that buys: immunity to clock offset, not to a paused holder.
The renew loop converges on the same shape everywhere because the arithmetic is the same: renew comfortably before expiry, and give up before someone else can have taken over. Kubernetes defaults to a 15-second lease with a 10-second renew deadline [3]; Patroni loops every 10 seconds against a 30-second TTL [11]; CockroachDB's original per-range leases lasted 9 seconds and renewed at 7.2 [10]. The client-go comment states the design consequence plainly: tolerance to clock-rate skew is exactly the ratio of LeaseDuration to RenewDeadline, so the person setting two YAML fields is choosing how fast one node's clock may run relative to another before safety is gone [1].
The loss handler is where implementations diverge most, and section 3
covers the argument. The fencing check is the component with three names
and one meaning. etcd's docs draw the genealogy in a single paragraph: Chubby called it a
sequencer, Kleppmann called it a fencing token, etcd calls it the revision number, and
ZooKeeper's zxid plays the same role [9]. CockroachDB implements
it as a per-node epoch: a lease is valid only while its epoch matches the node's liveness
record, renewal is a conditional put, and an epoch bump revokes every lease the node held
at once [10]. Kafka implements it broker-side: the latest
producer with a transactional id "fences" all previous instances, which then receive a
fatal ProducerFencedException [17]. The pattern in
who has fencing is hard to miss: it exists where one project owns both the lock and the
resource. Where the resource is yours (a Postgres table, an S3 bucket, a third-party API),
the token is offered by etcd and ZooKeeper, surfaced by neither client-go nor GitLab's
ExclusiveLease [1][19], and enforced by
nobody but you.
Four forks in the road, each with a recorded argument behind it and a condition that flips the answer.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Lock store for correctness work | Quorum store (ZooKeeper, etcd, Consul) | Single Redis with replica failover | Async replication loses the key on promotion; the vendor's own page marks it "SAFETY VIOLATION!" | Redis docs |
| Guarding a resource you own | Token or epoch checked in the write path | Trusting lease possession | "The lock feature of etcd itself cannot be used for protecting external resources" | etcd why.md |
| Guarding a resource you cannot change | Exit on loss, watchdog behind it, lock-delay in front | Graceful async cleanup | Demotion "may fail to happen" under OOM, pauses or slow shutdown; Chubby-style delay covers stragglers "while not a bulletproof method" | Patroni, Consul |
| Who wins an election | An arbiter picks the candidate (coordinated election) | Symmetric race, first renewer wins | Racing elections pick arbitrary versions: ~25% skew violation per 3-node upgrade, near-certainty on rollback | KEP-4355 |
| When the lock store is unreachable | Demote every primary (safety first) | Keep running on stale leases | "It is impossible to distinguish" store outage from partition; the failsafe carve-out requires every member to acknowledge the primary | Patroni failsafe |
| Waking up waiters | Watch one predecessor node | Broadcast on release | Waking every waiter stampedes the store: "this is important to avoid the herd effect" | ZooKeeper recipe |
Chubby's sequencer, Kleppmann's fencing token and etcd's revision number are the same idea, and etcd's documentation says so explicitly, crediting all three in one paragraph. CockroachDB's liveness epoch and Kafka's producer epoch are the same idea again, shipped inside the write path. If a design review uses different words for these, it is one concept: a number that only goes up, checked by the thing being written to.
The incidents in this corpus group into three classes: the paused holder that outlives its lease, the lock service that develops the same disease itself, and the detector that cannot tell split brain from Tuesday.
"etcd's locks aren't actually locks: like all 'distributed locks' they cannot guarantee mutual exclusion when processes are allowed to run slow or fast, or crash, or messages are delayed, or their clocks are unstable, etc." Kyle Kingsbury (aphyr), etcd issue #11457, December 2019 [7]
What is missing from this catalogue matters as much as what is in it. Nobody in this corpus published an incident where a fencing check existed and failed; the losses happen where the check is absent or where the lease machinery itself regressed. And no public record here describes application data corrupted through GitLab's unfenced ExclusiveLease pattern, despite a decade of production use across thousands of callers. Read that second absence carefully: it is either evidence that TTL leases plus idempotent job design absorb the zombie window in practice, or evidence that this failure is silent and unattributed when it happens. Both readings argue for the same design: make double execution cheap, because you will not reliably detect it.
The TTLs the industry actually runs, and the measured costs of getting this wrong. Every value carries its source and date.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Leader lease duration / renew deadline / retry | 15 s / 10 s / 2 s | Kubernetes | Defaults for every control-plane component election | 2026-09 | defaults.go |
| Tolerated clock-rate skew | LeaseDuration ÷ RenewDeadline | Kubernetes | 60 s / 30 s tolerates one clock running 2× another | 2026-09 | client-go comment |
| Default lock-session TTL | 60 s | etcd | concurrency.Session behind Mutex and Election | 2026-09 | session.go |
| Leader TTL / loop / watchdog margin | 30 s / 10 s / 5 s | Patroni | Watchdog fires 5 s before TTL; set margin to -1 for ttl÷2 if you need the hard guarantee | 2026-09 | watchdog.rst |
| Post-invalidation lock-delay | 15 s default, 0–60 s | Consul | Chubby-inspired stall before a lost lock is re-acquirable | v1.8 docs | sessions doc |
| Per-range lease / renewal point | 9 s / 7.2 s | CockroachDB | The pre-2016 design; renewal traffic at 10,000 ranges motivated epoch leases | 2016-02 | RFC 20160210 |
| Redlock reference shape | 5 masters, 10 s validity, 5–50 ms per-node timeout | Redis | Plus a crashed node must stay down longer than max TTL | 2026-09 | Redis docs |
| Writes lost trusting the lock alone | 10 of 42 in 60 s | etcd (Jepsen test) | Healthy-looking cluster, process pauses, no fencing | 2019-12 | etcd #11457 |
| Skew violation odds, racing election | ~25% per upgrade | Kubernetes | 3-node control plane; "almost a certainty" on rollback | 2024 | KEP-4355 |
| Exposure window of the two-leaseholder bug | v22.2 → v24.1.0 | CockroachDB | Roughly two years of releases; fixed in three maintenance lines | 2024-10 | a131639 |
The defaults are measured from source and will drift with releases; the Jepsen loss figure is from a deliberately adversarial test, not steady-state operation; the 25% is the KEP authors' own estimate for one specific topology. Two quantities nobody in this corpus publishes: the real-world frequency of zombie-holder windows in healthy fleets, and the p99 duration of stop-the-world pauses in the runtimes doing the holding. Your GC logs are the only source for the second one, and it is the number your TTL has to beat.
Every source behind this page, graded. The full ledger with copied quotes
ships alongside as sources.md. This corpus is repository artefacts only; the
absence of blog, paper and talk tiers is a network constraint of the research session,
not a judgement of the material.
A lease expiration that "can move back in time" during promotion let one range have two leaseholders; committed multi-range writes "could be irrecoverably lost" across two years of releases.
A production report: an etcd leader blocked in fdatasync was replaced but kept revoking leases as if primary, deposing every lock holder at once. Fixed by "Ignore old leader's leases revoking request" in 3.6.
Five dated incident records for "more than one postgres instance in read-write mode" on the production Patroni fleet. At least three of the recent four were false positives from provisioning and monitoring-label changes.
"This implementation does not guarantee that only one client is acting as a leader (a.k.a. fencing)", plus the exact clock-rate-skew tolerance your two timeout fields buy you.
A production user's report from the library's first year: the deposed leader never received OnStoppedLeading and kept acting as primary for a Postgres HA system.
A 10,000-candidate stress test recorded overlapping leadership terms ("[DIRTY RECORD]") in the same library, at the concurrency extreme.
Merged in three days. Before it, a deposed scheduler kept binding pods from a stale cache; after it, loss of lease is process death, matching the controller manager.
Proposed letting a deposed leader stay alive and rejoin. NAKed: standbys exist, restarts re-acquire, and "without fencing we need to exit(1) as fast as possible".
The recorded case against symmetric racing elections: ~25% chance per 3-node upgrade of a version-skew violation, near-certain on rollback, plus lease flip-flop. An arbiter now picks the candidate; beta at v1.33.
The lock RPC could return success on an expired lease; and the general claim that no distributed lock guarantees exclusion, with 10/42 writes lost in a 60-second test.
etcd answered Jepsen with documentation, a genealogy (Chubby's sequencer, Kleppmann's fencing token, etcd's revision) and an executable GC-pause demo, keeping the API and stating "the lock feature of etcd itself cannot be used for protecting external resources".
Two programs and an etcd cluster reproduce the paused-holder race on demand, then show the storage-side version check rejecting the stale write.
A production fencing design in full: a monotonic per-node epoch anchors every lease, renewal is a conditional put, and an epoch bump revokes everything the node held. Motivated by renewal traffic at 10,000 ranges.
The layered defence for a resource that cannot fence: demote on lease-update failure, kernel watchdog for when demotion itself fails, and a documented residual window in the default margin. Failsafe mode trades some of this for availability, gated on unanimous member acknowledgement.
Locks are "advisory mechanisms"; TTL expiry is a lower bound that can arrive late; and the lock-delay stalls re-acquisition for 15 seconds by default so a still-live former holder can notice, "while not a bulletproof method".
Documents the failover race as a safety violation, specifies Redlock's five-master design and timing assumptions, requires crashed nodes to stay down past the TTL, and links both sides of the 2016 Kleppmann/antirez debate.
Sequence nodes plus a watch on the single predecessor: no polling, no timeouts on the client path, and one wake-up per release, "important to avoid the herd effect".
Leases with no absolute timestamps anywhere: an acquirer watches the record's GUID stay unchanged for a full lease duration on its own clock before expiring it.
Broker-side fencing in one sentence of javadoc: the latest producer with a transactional id fences all previous instances, whose next request fails fatally.
Long GC pauses ("GC of Death") expire the ZooKeeper session, and the region server then shuts itself down "by design ... so that it stops serving data that may already be assigned elsewhere".
The application-tier pattern at scale: a Redis key with TTL, a UUID guard on release, no fencing token surfaced to callers, used across the codebase as "a cheap alternative to using SQL queries and updates".
component-base defaults (15 s / 10 s / 2 s) and etcd's defaultSessionTTL (60 s): the TTLs most of the industry actually runs, measured from source rather than quoted from docs.
Six rungs from a toy lease to the operational surface. The crossing from toy to real is rung 3, and etcd's maintainers have already written it for you.
One Redis or etcd instance, a key with a 10-second TTL, two workers competing with set-if-absent plus a unique holder id, release guarded by that id (the shape of GitLab's ExclusiveLease).
Done when: two workers run for an hour and the log shows exactly one holder at a time. Teaches: acquire, renew, guarded release.
SIGSTOP the holder for longer than the TTL, then SIGCONT. Watch the second worker acquire while the first resumes and finishes its critical section anyway. Log both writes with timestamps.
Done when: you have a log line proving two holders wrote inside one guarded window. Teaches: the pause-expiry race is trivial to hit on demand, and invisible unless you log for it.
Run etcd's own executable example: a storage process that records the highest lock revision seen and rejects lower ones, with a GC-pause client. Then rebuild rung 2 with a token column and a conditional update in your own store.
Done when: the resumed zombie's write is rejected with a version mismatch, as in the example's output. Teaches: exclusion lives at the resource; the lock only queues.
Wire client-go leader election (or an etcd session) so that lease loss calls exit, not a cleanup routine. Kill the elected leader's network and measure the gap between old-leader exit and new-leader start against LeaseDuration.
Done when: failover time matches the lease arithmetic and no work happens between depose and exit. Teaches: why PR #81306 chose Fatalf, and what the 15 s default costs in downtime.
Three-node etcd; pause the leader's disk (or the process) mid-load and watch lease behaviour across the election, replaying issue #15247's shape. Add jitter to session TTLs and an alert on cluster-wide revocation rate.
Done when: you can state the blast radius of a store leader stall in holders deposed per second. Teaches: the store is a lease holder too, with the same disease.
Add the double-primary detector (count of writable holders per lock, GitLab's observable), make it provisioning-aware, and write the runbook: what pages, what auto-remediates, and which false-positive classes you already know about.
Done when: a cluster rebuild does not page anyone and a forced double-primary pages within one TTL. Teaches: detection is a design problem of its own, and the GitLab record shows where it goes wrong.
The queries that found this material, adapted for reuse. The common trick: search for the admission, not the feature.
"does not guarantee that only one client is acting""cannot be used" mutual exclusion lease site:github.com"not a bulletproof" OR "advisory" lock sessions docsfencing token revision zxid epoch sequencergitlab.com/gitlab-com/gl-infra/production issues: "split brain"repo:etcd-io/etcd is:issue lease revoked leader stuck<vendor>/docs advisories "lease" OR "leaseholder"site:github.com issues "still believes it is the leader"repo:kubernetes/kubernetes is:pr is:closed is:unmerged leaderelection"exit(1)" OR Fatalf "leader" lost lease pull requestjepsen is:issue repo:<org>/<repo>raw.githubusercontent.com/<org>/<repo>/<old tag or PR commit>/<path>repo:<org>/<repo> path:docs/RFCS lease