Portfolio ↗
Evidence ledger 22 sources Checked 19 Sep 2026

Evidence ledger

One row per claim in When two hold the lock: what the repositories admit about distributed mutual exclusion: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.

Field guide: When two hold the lock: what the repositories admit about distributed mutual exclusion. Research date: 2026-09-19. Every artefact below was fetched during the session, either with a direct curl to raw.githubusercontent.com / gitlab.com, with git clone --depth=1 --filter=blob:none, or as the rendered github.com page for issues and pull requests. Quotes are copied, not paraphrased.

A note on the shape of this corpus. The session's network policy reached code hosts only (github.com, raw.githubusercontent.com, gitlab.com, pkg.go.dev). Engineering blogs, conference talks, papers and postmortems published on company blogs were unreachable — including Martin Kleppmann's and Salvatore Sanfilippo's 2016 posts, the Chubby paper, the Jepsen etcd report, and the Roblox, GitHub and Cloudflare outage write-ups. This ledger therefore contains what the repositories themselves record: code comments, project documentation, issues, pull requests (merged and rejected), design records, advisories, and GitLab's public production incident tracker. Where the guide needs the blog-side argument it cites the repository artefact that quotes, links or implements it, and says so. Postmortem-tier rows are dated incident records: a vendor advisory disclosing production data loss, an issue reporting a production outage, and GitLab's incident-review issues.

# Org Title Tier Published Checked URL Claim taken from it Supporting quote or figure
1 Kubernetes client-go leaderelection package comment source in tree since 2015; current master 2026-09-19 https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go The standard Kubernetes leader election states, in its own package comment, that it does not fence "This implementation does not guarantee that only one client is acting as a leader (a.k.a. fencing)."
2 Kubernetes client-go leaderelection package comment source current master 2026-09-19 https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go Safety depends on relative clock rate, and the tolerated ratio is set by configuration "the implementation is tolerant to arbitrary clock skew, but is not tolerant to arbitrary clock skew rate" and "if a user wanted to tolerate some nodes progressing forward in time twice as fast as other nodes, the user could set LeaseDuration to 60 seconds and RenewDeadline to 30 seconds."
3 Kubernetes component-base leader-election defaults source current master 2026-09-19 https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/component-base/config/v1alpha1/defaults.go Default control-plane lease numbers: LeaseDuration 15 s, RenewDeadline 10 s, RetryPeriod 2 s obj.LeaseDuration = metav1.Duration{Duration: 15 * time.Second} / RenewDeadline ... 10 * time.Second / RetryPeriod ... 2 * time.Second
4 Kubernetes Issue #23731 "Split-Brain bug in leaderelection client" source 2016-04-01 2026-09-19 https://github.com/kubernetes/kubernetes/issues/23731 A deposed leader kept acting because the loss callback never fired — reported from a Postgres HA setup built on the library "postgres-system-1 used to be the leader but lost the lease. However, it still believes it is the leader since OnStoppedLeading() is never called"
5 Kubernetes Issue #67651 "Client-go leader election can potentially split brain" source 2018-08-21 2026-09-19 https://github.com/kubernetes/kubernetes/issues/67651 Under a 10,000-candidate stress test the library recorded overlapping leadership terms "at the extreme cases, the leaderelection pkg doesn't working correctly as expected"; test output marked "[DIRTY RECORD]" where terms overlapped
6 Kubernetes PR #81306 "Scheduler should terminate on loosing leader lock" source merged 2019-08-15 2026-09-19 https://github.com/kubernetes/kubernetes/pull/81306 The fix for a deposed scheduler that kept scheduling was to make it die, matching the other control-plane components Without the change the deposed scheduler kept "watching pod, node objects and allowing the pod to be scheduled", causing "multiple bind failures as the individual scheduler will have a stale cache"
7 Kubernetes PR #54605 "fix leaderelection: renew lease shouldn't exit, follower maybe leader again" source closed unmerged 2017-11-22 2026-09-19 https://github.com/kubernetes/kubernetes/pull/54605 The opposite design — letting a deposed leader rejoin in-process — was rejected; exiting is the stated safety mechanism in the absence of fencing Reviewer NAK: instances can "reacquire a lease" after restart, users can "have (n) stand-bys", and "without fencing we need to exit(1) as fast as possible"
8 Kubernetes KEP-4355 Coordinated Leader Election adr kep.yaml creation-date: 2023-14-05 (sic); stage beta, latest-milestone v1.33 2026-09-19 https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/4355-coordinated-leader-election/README.md Symmetric racing elections pick an arbitrary winner, which violates version-skew policy during upgrades; the KEP adds an arbiter that chooses the candidate "For 3 node control plane upgrade, there is about a 25% chance of a new version of the controller running while old versions of the apiserver are active, resulting in a skew violation."; "For rollback, it is almost a certainty that skew will be violated."; the lease may be "flip-flopping between old and new"
9 etcd Issue #11456 "Locks return without checking whether the lease is still held" (aphyr) source 2019-12-16 2026-09-19 https://github.com/etcd-io/etcd/issues/11456 The lock RPC could return success after the caller's lease had already expired "while waiting for the previous lock holder to release the lock, the client does not check the current lease status. So when the lock operation returns, the lease might be already expired."
10 etcd Issue #11457 "Document that locks aren't really locks" (aphyr) source 2019-12-16 2026-09-19 https://github.com/etcd-io/etcd/issues/11457 The Jepsen finding, filed as an issue: no distributed lock guarantees exclusion under pauses, delays or clock trouble "etcd's locks aren't actually locks: like all 'distributed locks' they cannot guarantee mutual exclusion when processes are allowed to run slow or fast, or crash, or messages are delayed, or their clocks are unstable, etc."
11 etcd Issue #11457, measured loss source 2019-12-16 2026-09-19 https://github.com/etcd-io/etcd/issues/11457 Measured impact of trusting the lock alone, in a healthy-looking test "This 60-second test lost 10/42 successfully completed writes"; recommended remedy: users "must carefully couple a fencing token (e.g. the etcd lock key revision number) to any systems they interact with in order to preserve exclusion boundaries."
12 etcd PR #11490 "RFC Documentation: enhance description of lock and lease" adr merged 2020-03-05 2026-09-19 https://github.com/etcd-io/etcd/pull/11490 The project answered the Jepsen finding with documentation and an executable counter-example, not an API change PR closes #11457; adds Documentation/learning/lock/ with client/storage programs demonstrating false expiry under GC pause
13 etcd Documentation/learning/why.md (at commit db9bbe8, merged by PR #11490) adr 2020-03-05 2026-09-19 https://github.com/etcd-io/etcd/blob/db9bbe8ba37ac89ab1ad7e6e5a734ac9ef6c8293/Documentation/learning/why.md The project's own statement that its lock is an optimisation, named for historical reasons, and unsafe for external resources without resource-side validation "the lock APIs cannot be used as mutual exclusion mechanism. The APIs are called lock because of [the historical reason][chubby]"; "Actually, the lease mechanism itself doesn't guarantee mutual exclusion."; "The lock feature of etcd itself cannot be used for protecting external resources."
14 etcd Documentation/learning/why.md, fencing genealogy adr 2020-03-05 2026-09-19 https://github.com/etcd-io/etcd/blob/db9bbe8ba37ac89ab1ad7e6e5a734ac9ef6c8293/Documentation/learning/why.md One mechanism, three names: Chubby's sequencer, Kleppmann's fencing token, etcd's revision number "In [the paper of Chubby][chubby], the concept of sequencer is introduced. We interpret that sequencer is an almost same to the combination of revision number and lease ID of etcd." and "Martin Kleppmann introduced the idea of fencing token. The authors interpret that fencing token is revision number in the case of etcd."
15 etcd Documentation/learning/lock/README.md (at db9bbe8) source 2020-03-05 2026-09-19 https://github.com/etcd-io/etcd/blob/db9bbe8ba37ac89ab1ad7e6e5a734ac9ef6c8293/Documentation/learning/lock/README.md The executable reproduction: a GC-paused client's write is rejected by version validation "Generally speaking, a lease-based lock service cannot provide mutual exclusion to processes."; demo output: "failed to write to storage: error: given version (4703569812595502721) differ from the existing version (4703569812595502727)"
16 etcd Issue #15247 "All leases are revoked when the etcd leader is stuck…" postmortem 2023-02-06 2026-09-19 https://github.com/etcd-io/etcd/issues/15247 Production incident report: a leader stuck on slow fdatasync mass-revoked leases, deposing every lock holder at once; the lock service had its own stale-leader bug "The etcd pod use portworx storage for db saving, we've found when rebooting the portworx node, it will cause etcd to be stuck in fdatasync"; the old leader "continued revoking all leases after transitioning to follower"; fixed by PR #16822 "Ignore old leader's leases revoking request"
17 etcd client/v3/concurrency/session.go source current main 2026-09-19 https://github.com/etcd-io/etcd/blob/main/client/v3/concurrency/session.go Default lock-session lease TTL is 60 s const defaultSessionTTL = 60
18 CockroachDB Technical Advisory a131639 postmortem 2024-10-08 2026-09-19 https://github.com/cockroachdb/docs/blob/main/src/current/advisories/a131639.md In production releases v22.2–v24.1.0, a lease-expiry regression during slow disk I/O produced two simultaneous leaseholders and irrecoverably lost committed writes "the expiration time for the lease transferred to the node with the slow stores can move back in time during this promotion … the range can have two leaseholders."; "it is possible for some writes in a transaction that straddle multiple ranges to be lost … these writes could be irrecoverably lost."
19 CockroachDB RFC 20160210 "Node-level mechanism for refreshing range leases" adr 2016-02-10 2026-09-19 https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20160210_range_leases.md A production fencing design: leases carry a monotonically increasing epoch checked on every command; renewal is a conditional put; an epoch bump revokes every lease the node held "Epoch
20 CockroachDB RFC 20160210, renewal-traffic numbers adr 2016-02-10 2026-09-19 https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/20160210_range_leases.md Per-resource expiry leases were replaced because renewal traffic scales with resources, not nodes "Range leases have a moderate duration (currently 9 seconds)"; "the lease is renewed before it expires (currently, after 7.2 seconds)"; motivating example "a table with 10,000 ranges"
21 GitLab Production incident 2022-04-08 (#6795) "Split Brain: too many postgres databases … in read-write (primary) mode" postmortem 2022-04-08 2026-09-19 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6795 The double-primary alert has fired on the production Patroni fleet since at least 2022; this occurrence was labelled a configuration change at severity 4 Issue labels: RootCause::Config-Change, Service::Patroni, severity::4
22 GitLab Production incident 2025-10-25 (#20767) postmortem 2025-10-25 2026-09-19 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/20767 A split-brain alert with no split brain: cluster provisioning looked identical to failure "The alert was triggered by the creation of a new Postgres cluster (v17), not by an actual problem with database traffic or customer data."
23 GitLab Production incident 2026-07-13 (#22502) postmortem 2026-07-13 2026-09-19 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22502 Another false-positive split-brain page, this time from monitoring label grouping after a rolled-back upgrade "this was a false-positive alert caused by incorrect monitoring grouping"; "We merged a monitoring label update so the leftover Postgres 18 node no longer triggers the incorrect alert grouping"
24 GitLab Production incident 2026-06-27 (#22394) postmortem 2026-06-27 2026-09-19 https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22394 The detector in production words: more than one read-write instance is the observable of split brain "The patroni-sec PostgreSQL cluster has more than one instance in read-write mode, indicating a split-brain condition."
25 GitLab lib/gitlab/exclusive_lease.rb source current master 2026-09-19 https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/exclusive_lease.rb A large production codebase's general-purpose lock is a Redis TTL lease with a UUID-guarded release and no fencing token surfaced to callers "This class implements an 'exclusive lease'. We call it a 'lease' because it has a set expiry time. We call it 'exclusive' because only one caller may obtain a lease for a given key at a time."; cancel script: if redis.call("get", key) == uuid then redis.call("del", key)
26 Zalando (Patroni) docs/watchdog.rst vendor current master 2026-09-19 https://github.com/zalando/patroni/blob/master/docs/watchdog.rst Stopping the deposed primary can itself fail, so the fallback is a kernel watchdog reset — and even that has a stated hole "Patroni will try to achieve this by stopping PostgreSQL when leader lock update fails for any reason. However, this may fail to happen due to various reasons"; "By default Patroni will set up the watchdog to expire 5 seconds before TTL expires."; "This results in a window of time where watchdog will not trigger before leader key expiration, invalidating the guarantee."
27 Zalando (Patroni) docs/dcs_failsafe_mode.rst vendor current master 2026-09-19 https://github.com/zalando/patroni/blob/master/docs/dcs_failsafe_mode.rst Losing the lock store demotes healthy primaries by design; the escape hatch requires unanimous acknowledgement from every member "the node is allowed to run Postgres as the primary only if it can update the leader lock in DCS. In case the update of the leader lock fails, Postgres is immediately demoted and started as read-only."; "In general, it is impossible to distinguish between these two from a single node, and therefore Patroni assumes the worst case - network partitioning."; "may continue to run as the primary if all replicas acknowledge it. If one of the members doesn't respond, the primary is demoted."
28 HashiCorp Consul sessions.mdx (v1.8.0 tag) vendor tagged 2020-06 (doc text older) 2026-09-19 https://github.com/hashicorp/consul/blob/v1.8.0/website/pages/docs/internals/sessions.mdx Consul's locks are advisory; TTL expiry is a lower bound; the lock-delay safeguard is inherited from Chubby and admits it is not bulletproof "This ID can be used with the KV store to acquire locks: advisory mechanisms for mutual exclusion."; "Consul will not expire the session before the TTL is reached, but it is allowed to delay the expiration past the TTL"; "this is a safeguard inspired by Google's Chubby. … While not a bulletproof method, it does avoid the need to introduce sleep states into application logic"; default delay 15 seconds, range 0–60
29 Apache ZooKeeper recipes.md (branch-3.9) vendor current branch-3.9 2026-09-19 https://github.com/apache/zookeeper/blob/branch-3.9/zookeeper-docs/src/main/resources/markdown/recipes.md The canonical sequence-node lock recipe: no timeouts on the client path, wake exactly one waiter "Call getChildren( ) on the lock node without setting the watch flag (this is important to avoid the herd effect)"; "The removal of a node will only cause one client to wake up since each node is watched by exactly one client."; "There is no polling or timeouts."
30 Redis distributed-locks.md (redis/docs) vendor current main 2026-09-19 https://github.com/redis/docs/blob/main/content/develop/clients/patterns/distributed-locks.md The failover race that motivates Redlock, stated as a safety violation by the vendor itself "The master crashes before the write to the key is transmitted to the replica. The replica gets promoted to master. Client B acquires the lock to the same resource A already holds a lock for. SAFETY VIOLATION!"
31 Redis distributed-locks.md, the efficiency concession vendor current main 2026-09-19 https://github.com/redis/docs/blob/main/content/develop/clients/patterns/distributed-locks.md The vendor's own text concedes the efficiency/correctness split and carries both sides of the 2016 debate "Sometimes it is perfectly fine that, under special circumstances, for example during a failure, multiple clients can hold the lock at the same time. If this is the case, you can use your replication based solution."; "Martin Kleppmann [analyzed Redlock here]. A counterpoint to this analysis can be [found here]."
32 Redis distributed-locks.md, timing assumptions and restart rule vendor current main 2026-09-19 https://github.com/redis/docs/blob/main/content/develop/clients/patterns/distributed-locks.md Redlock's safety rests on bounded clock drift and on crashed nodes staying down past the TTL "The algorithm relies on the assumption that … the local time in every process updates at approximately at the same rate"; "mutual exclusion … is guaranteed only as long as the client holding the lock terminates its work within the lock validity time"; "make an instance, after a crash, unavailable for at least a bit more than the max TTL we use"; example numbers: N=5 masters, 10 s auto-release, 5–50 ms per-node timeouts
33 AWS amazon-dynamodb-lock-client README vendor repo published 2017; current master 2026-09-19 https://github.com/awslabs/amazon-dynamodb-lock-client/blob/master/README.md A lease design that deliberately avoids absolute clocks: expiry is observed elapsed time on the observer's own clock "The lock client never stores absolute times in DynamoDB -- only the relative 'lease duration' time is stored"; "even if two different machines disagree about what time it is, they will still avoid clobbering each other's locks."
34 Apache Kafka ProducerFencedException javadoc source current trunk 2026-09-19 https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/errors/ProducerFencedException.java Fencing shipped at the resource: the broker rejects the older holder of a transactional id "It is only possible to have one producer instance with a transactional.id at any given time, and the latest one to be started 'fences' the previous instances so that they can no longer make transactional requests."
35 Apache HBase troubleshooting doc (hbase-website tree, master) vendor current master 2026-09-19 https://github.com/apache/hbase/blob/master/hbase-website/app/pages/_docs/docs/_mdx/%28multi-page%29/troubleshooting.mdx Self-fencing by process death is a documented design decision, triggered by GC pauses long enough to expire the ZooKeeper session "long pauses in all threads including the Juliet Pause aka 'GC of Death'"; "By design, we shut down any node that isn't able to contact the ZooKeeper ensemble after getting a timeout so that it stops serving data that may already be assigned elsewhere."