# Architecture Decision Record

*Distributed Lock Service · Solution Architecture v1.0 · Platform Architecture · 2026-09 · 26 views · open source, fully on-premises*

The argument these decisions serve is summarised in the [Architecture One-Pager](architecture-one-pager).

Twenty-four decisions make this architecture. Everything else on the twenty-six views is convention, and convention needs no defending. Each record opens with the question that forced a decision and the context that makes it hard, states the decision so it can be checked, then shows how it is realised on hardware the operator owns: which package, configured how. It weighs the credible alternatives, including the ones that are right for a different organisation, says what the choice buys and what it costs, names the conditions that would flip it, and closes with two things that outlast this system: why the decision should still hold in ten years, and the lesson that transfers. Decisions are grouped into seven areas; use the filter to read one at a time.

> **Status of this document.** This is a design, not a report on a running system. Latency and throughput figures are targets and stated assumptions drawn from the requirement, to be replaced by benchmark numbers from the operator's own hardware before build. Every component is open source and runs on-premises; nothing depends on a public cloud service or a vendor-hosted endpoint. Where a record relies on a behaviour of a specific etcd release, the release gate on view 20 is what verifies it, and the record says so rather than trusting a changelog. The seven open questions in the requirement are each answered by a record here: Q1 by ADR-01, Q2 by ADR-16, Q3 by ADR-11, Q4 by ADR-12, Q5 by ADR-08, Q6 by ADR-14 and Q7 by ADR-07.

## How to read a record

- **Question:** The forcing question: why a decision was needed at all.
- **Context:** The requirement, the scale and the constraint that make it hard.
- **Decision:** What this architecture does, stated so it can be checked.
- **How it works on-premise:** The concrete mechanism: which package, configured how, on whose hardware.
- **Options weighed:** Chosen, rejected, deferred, or right elsewhere, with the reason for each.
- **Consequences:** What the choice buys and what it costs, both kept visible.
- **Choose differently when:** The conditions that would flip the decision for your system.
- **Why it holds up over time:** What keeps the decision right as scale, staff and technology change.
- **Lesson:** The principle that transfers beyond this platform.

## Decision map

**The guarantee**: What the service promises, what it refuses to promise, and how the token keeps that promise across disasters.

- ADR-01 · Promise ordered grants everywhere; claim mutual exclusion only for fenced classes
- ADR-02 · The fencing token is the epoch plus the etcd revision of the granting transaction
- ADR-03 · Bind the epoch to the etcd cluster ID and seal a restored cluster until the epoch advances
- ADR-04 · Fail closed on quorum loss; fail open only for declared efficiency classes, marked unprotected

**Consensus and topology**: The store, where it runs, how many of it there are, and why none of it spans regions.

- ADR-05 · etcd as the lock log
- ADR-06 · Dedicated etcd clusters on dedicated nodes, never the Kubernetes control plane's
- ADR-07 · Two cluster tiers, coord and entity, chosen by the lock class
- ADR-08 · Regional quorums only; no stretched quorum until a global resource is named and priced
- ADR-09 · Home entity keys by rendezvous hashing, and fence every move with an epoch advance

**Leases and holders**: Sessions, TTLs and what a holder does in the dark.

- ADR-10 · A session is one etcd lease per identity and TTL; every hold attaches to it
- ADR-11 · TTL is chosen per class within a platform range, with its renewal cost printed in review
- ADR-12 · Stop starting work at the first failed keepalive; fence at a deadline measured from send time
- ADR-13 · Re-entrancy is counted in the client library; the log sees only the outermost acquire and release

**Contention**: Queues, fairness and why a notification never grants anything.

- ADR-14 · In-memory FIFO on a key-affine arbiter by default; durable waiter keys as a class option
- ADR-15 · A watch notification triggers an attempt, never a grant

**Enforcement at the resource**: Where two writers are actually kept apart, and what happens where they are not.

- ADR-16 · The resource's own conditional write holds the high-water mark; validation API is the fallback
- ADR-17 · Ship a reference fence and a pause test kit; declare fenced only on a passing test

**Access and identity**: Why there is a service at all, who may lock what, and who may break a lock.

- ADR-18 · A stateless Go arbiter behind Envoy, rather than a thick client straight to etcd
- ADR-19 · SPIFFE workload identity, and authorisation as prefix grants rather than a policy engine
- ADR-20 · Force-release requires two operators, a reason, and a delete-only etcd role

**Evidence and operations**: Audit, telemetry, release gating and configuration.

- ADR-21 · Audit is tailed from etcd revisions into ClickHouse, never dual-written by the arbiter
- ADR-22 · Metrics by namespace and class only; tokens never in telemetry
- ADR-23 · No release without a fault-injected history checked for double grants
- ADR-24 · Lock classes and grants are reviewed CRDs delivered by Argo CD

## Technology by capability

Every capability on the views, the package that provides it, the credible alternative, and the record that justifies the choice. Everything here runs on the operator's own hardware.

| Capability | Choice | Origin | Credible alternative | Why this one | Record |
|---|---|---|---|---|---|
| Lock log (consensus store) | etcd 3.6 line, dedicated clusters | Open source | Apache ZooKeeper; Consul; custom Raft on hashicorp/raft | Leases revoked by committed entry, multi-key transactions, watch from revision, and the most heavily exercised Raft implementation in production | ADR-05 |
| Fencing token | (epoch, etcd revision) packed in int64 | This design | Per-key counter key; etcd lease ID | Minted by the grant commit itself, never reused, no extra state per key | ADR-02 |
| Restore safety | Epoch record bound to etcd cluster ID | This design | Runbook step; wall-clock epoch | A restore always changes the cluster ID, so the seal cannot be skipped | ADR-03 |
| Cluster topology | Coord tier (5) + entity shards (3 each), per region | This design | One large cluster; stretched multi-region quorum | Isolates leader-election locks from entity churn; writes scale by shards, not members | ADR-07 |
| Key homing | Rendezvous hashing with versioned shard map | This design | Consistent-hash ring; manual range assignment | Minimal key movement on shard add; each move fenced by epoch advance | ADR-09 |
| Session and renewal | One etcd lease per identity and TTL | Open source | One etcd lease per lock; application-level heartbeat keys | Renewal cost per session, batching by construction | ADR-10 |
| Lock arbiter | Go service using etcd clientv3, gRPC | This design | Thick client straight to etcd; Java service | Same client library etcd's own robustness tests use; small static binaries; one place for class rules and quotas | ADR-18 |
| Client protocol | gRPC + protobuf, buf breaking checks | Open source | REST/JSON; raw etcd API | Streaming keepalive and watch on one connection; contract enforced in CI | ADR-18 |
| Fan-in tier | Envoy, ring-hash on lock key | Open source | HAProxy; NGINX; client-side load balancing | gRPC-native, SPIFFE-aware mTLS, per-identity local rate limits, key-affine routing for queues | ADR-18 |
| Workload identity | SPIRE issuing X.509 SVIDs | Open source | cert-manager with a private CA; Kubernetes service account tokens | Attested identity, one-hour certificates, identity survives outside Kubernetes | ADR-19 |
| Operator identity | Keycloak OIDC with MFA | Open source | Directory groups via LDAP bind; Dex | Roles, MFA and a second-subject check without building an identity system | ADR-20 |
| Authorisation | Prefix grants in the class registry, evaluated in-process | This design | Open Policy Agent; SpiceDB | A longest-prefix match does not need a policy engine on a 40 ms path | ADR-19 |
| Wait queue | In-memory FIFO on key-affine arbiter; durable /q keys per class | This design | Durable queue for all; client polling | Free fairness for most classes, paid fairness where declared | ADR-14 |
| Resource fence | Store's own conditional write; reference for PostgreSQL and S3 conditional put | This design | Token-validation call on every write | Atomic with the write, no extra hop, no new dependency | ADR-16 |
| Audit capture | Revision tailer on etcd watch | This design | Arbiter emits events to Kafka | Captures expiries; no dual write; idempotent replay | ADR-21 |
| Audit store and archive | ClickHouse; MinIO with object lock | Open source | PostgreSQL; OpenSearch; Kafka tiered storage | Columnar scans over hundreds of millions of events; retention enforced by the store | ADR-21 |
| Metrics, traces, logs | Prometheus, Grafana, Alertmanager, OTel Collector, Tempo, Loki | Open source | VictoriaMetrics; Jaeger; OpenSearch | Already the platform standard; bounded label sets; redaction processor in the collector | ADR-22 |
| Release gate | Chaos Mesh + Porcupine linearizability checker | Open source | Jepsen; unit and integration tests only | Go-native, runs in the cluster, checks real histories on every change | ADR-23 |
| Configuration delivery | LockClass CRDs in Gitea, synced by Argo CD | Open source | Admin API writes to etcd; ConfigMaps | Reviewed by the resource owner; history, diff and rollback for free | ADR-24 |
| Build and supply chain | Gitea Actions, cosign, Harbor | Open source | Jenkins; GitLab CE | Signed images, admission-checked, on-premises end to end | ADR-23 |
| Disk encryption | LUKS with Clevis and Tang | Open source | Self-encrypting drives; manual passphrase | Unattended unlock inside the network, unreadable disk outside it | ADR-06 |

## The decisions, and the alternatives that lost

### The guarantee

*What the service promises, what it refuses to promise, and how the token keeps that promise across disasters.*

#### ADR-01 · Promise ordered grants everywhere; claim mutual exclusion only for fenced classes

**Status:** Accepted  ·  **Shown on views:** 01, 15, 17

*Does the service promise that two processes never act at once, or only that grants are ordered?*

**Context.** The requirement asks for correctness locks where two holders cause damage. But a holder paused past its lease cannot be stopped by any protocol, so exclusion can only be enforced where the write lands. Requiring every guarded resource to be fenced before a correctness lock can be created is the honest guarantee and blocks adoption on legacy stores. Promising only ordered grants ships everywhere and quietly leaves the strongest claim unbacked.

**Decision.** The service promises ordered, expiring grants for every class. It claims mutual exclusion only for classes declared fenced, and a class may be declared fenced only after the pause test passes against the real resource. A correctness class may be created as advisory, but only with a named risk owner and an expiry date, and it appears on a standing risk register until fenced or renewed on the record.

**How it works on-premise.** The LockClass CRD has required fields enforcement: fenced | advisory and, for advisory, riskOwner and waiverExpires. A validating admission webhook rejects a class that omits them. The class controller refuses to mark a class fenced unless a pause-test result object referencing that class and resource exists. ClickHouse produces the advisory register nightly from the class list and grant counts.

| Option | Verdict | Reasoning |
|---|---|---|
| Ordered grants everywhere, exclusion claimed only when fenced, advisory registered | Chosen | Ships to every team on day one without letting any of them believe they are protected when they are not. |
| Refuse correctness locks on any unfenced resource | Rejected | Honest, and it sends the teams with legacy stores back to a Redis key with an expiry, which is strictly worse. |
| Promise exclusion and document the pause caveat | Rejected | A caveat nobody reads is the industry's current state. It is how duplicate postings happen in systems that have a lock. |
| Refuse locks entirely and push conditional writes | Right elsewhere | Right for an estate where every store supports compare-and-set; many do, and ADR-16 pushes teams there first. |

**What it buys**

- The platform's claim is exactly as strong as the evidence for it
- Unfenced risk is visible to security and has a date on it
- Adoption is not blocked on the slowest resource team

**What it costs**

- A second concept for every owner to understand
- Waiver administration and quarterly review
- Some advisory classes will be renewed indefinitely, and the register makes that visible rather than preventing it

**Choose differently when.** Flip to refusing unfenced correctness locks when every guarded store in the estate supports a conditional write and the register has been empty for two quarters. At that point advisory correctness is only a way to skip work.

**Why it holds up over time.** The distinction between an ordered grant and enforced exclusion is a law of distributed systems, not a feature of this stack. It will be true of whatever store replaces etcd, and the register keeps the gap visible through every reorganisation.

> **Lesson.** Promise what you can prove, register what you cannot, and put a date on the second list.

#### ADR-02 · The fencing token is the epoch plus the etcd revision of the granting transaction

**Status:** Accepted  ·  **Shown on views:** 11, 14

*How is a 64-bit token minted that strictly increases per key, is never reused, and is committed with the grant?*

**Context.** The obvious design is a counter key per lock, incremented in the grant transaction. It works, and the counter must then live forever, because deleting it lets the next grant restart at one. With four million distinct keys a day, the counter table becomes the largest thing in the store and a compaction problem nobody planned. The token must also stay monotonic across restores, where any stored value can go backwards.

**Decision.** The token is epoch × 2^48 + R, where R is the etcd revision at which the grant transaction committed and the epoch occupies the top bits, kept below 2^15 so the value is positive in a signed 64-bit column. Revisions are cluster-wide, strictly increasing and never reused within a cluster's life, which is stronger than per-key monotonicity. Gaps are expected and carry no meaning.

**How it works on-premise.** The arbiter reads header.revision from the successful TxnResponse and packs it with the epoch it compared in the same transaction. No additional key is written. The proto carries the token as a fixed64 plus the decoded (epoch, revision) pair for diagnostics only; resources compare the packed integer.

| Option | Verdict | Reasoning |
|---|---|---|
| Epoch plus commit revision | Chosen | Minted by the commit itself, zero extra state, and a restore is handled once by the epoch rather than per key. |
| Per-key counter key incremented in the grant transaction | Rejected | Correct, but the counters can never be deleted and still need an epoch to survive a restore. |
| etcd lease ID as the token | Rejected | Lease IDs are not ordered with respect to grants on a key, and many holds share one session lease. |
| Hybrid logical clock timestamp | Rejected | Adds clock assumptions to the one property that must hold without them. |

**What it buys**

- No token state to compact or migrate
- Tokens from different keys are comparable, which simplifies debugging
- Header revision is already returned on every etcd response

**What it costs**

- Token values are large and non-contiguous, which surprises people expecting 1, 2, 3
- Token semantics are tied to one cluster's revision space; rehoming a key requires an epoch advance (ADR-09)

**Choose differently when.** Choose a per-key counter if the store behind the arbiter has no global monotonic commit index, for example a store that shards internally without exposing one.

**Why it holds up over time.** 2^48 revisions last about 297 years at 30,000 commits a second, per epoch. The packed format is frozen in the proto contract, so resources never need to change the width of the column they store it in.

> **Lesson.** Before adding state to produce an ordering, look for an ordering the system already maintains for its own reasons.

#### ADR-03 · Bind the epoch to the etcd cluster ID and seal a restored cluster until the epoch advances

**Status:** Accepted  ·  **Shown on views:** 13, 26

*How is a restored lock log prevented from issuing a token it has issued before?*

**Context.** Restoring a snapshot rolls revisions back to the snapshot point. Any grant made after the snapshot has handed a holder a token the restored cluster will issue again. The requirement says this must be impossible by construction, not by procedure. Restores happen during the worst hour of an operator's year, which is exactly when procedural steps get skipped.

**Decision.** A /sys/epoch record stores the current epoch and the etcd cluster ID it was minted for. Every grant compares the epoch key. The arbiter compares the cluster ID in every response header with the one in the epoch record and refuses to serve on mismatch. A restore always produces a new cluster ID, so a restored cluster is sealed until a single transaction sets epoch E+1 bound to the new ID and deletes every restored hold.

**How it works on-premise.** etcdutl snapshot restore generates a new cluster ID by design. The epoch-advance job is a Kubernetes Job gated by a two-person approval in the Admin API; it runs one Txn: compare epoch = E, put /sys/epoch = {E+1, newClusterID}, delete range /l and /q. New epochs are allocated from the coord cluster so no two clusters ever reuse one.

| Option | Verdict | Reasoning |
|---|---|---|
| Cluster-ID-bound epoch with arbiter seal | Chosen | The service cannot serve without the advance, so the advance cannot be forgotten. |
| Runbook step to bump the epoch after restore | Rejected | Correct on the day it is followed, and one skipped step is a duplicate token. |
| Epoch derived from wall-clock time at startup | Rejected | Puts a clock back into the one property designed to be independent of clocks. |
| Never restore; rebuild empty and let everyone reacquire | Right elsewhere | Acceptable if tokens are also epoch-advanced; the design effectively does this, keeping the snapshot for the shard map and class state. |

**What it buys**

- Token regression is impossible without deliberately editing the epoch key
- The seal is visible: arbiters report sealed status on every request
- Rehoming reuses the same mechanism

**What it costs**

- Every hold is released at once after a restore, producing a reacquisition storm
- A second person must be reachable during disaster recovery
- Relies on etcd continuing to assign a new cluster ID on restore, which the release gate checks

**Choose differently when.** Unnecessary on a store whose restore preserves a monotonic index beyond the snapshot, such as one with continuous log archiving to a point after the last committed entry. Such stores are rare and still benefit from the seal as a guard.

**Why it holds up over time.** The mechanism depends on one invariant, that a restored cluster is distinguishable from the original, and on nothing else. Any future store can supply an equivalent identity, and the arbiter check is ten lines.

> **Lesson.** Make the dangerous state unservable, not merely documented.

#### ADR-04 · Fail closed on quorum loss; fail open only for declared efficiency classes, marked unprotected

**Status:** Accepted  ·  **Shown on views:** 17, 26

*When no majority is reachable, does the service keep granting?*

**Context.** Correctness classes must never issue a second grant, so no grants and no renewals can happen without a quorum, and holders must self-fence as their deadlines pass. Efficiency classes, where two holders only waste money, would rather keep running. The requirement wants that difference declared and every open grant traceable afterwards.

**Decision.** Every class declares onQuorumLoss. Correctness classes may only declare closed. Efficiency classes may declare open, in which case an arbiter that cannot reach a majority grants from local memory with token zero, spools each grant to a local write-ahead file, and ships the spool to the audit store with unprotected = true on recovery. The MVP is closed for all classes.

**How it works on-premise.** The class admission webhook rejects enforcement: fenced or a correctness purpose combined with onQuorumLoss: open. The spool is a small append-only file on the arbiter pod's local persistent volume, shipped by the tailer's recovery step. The reconciliation list is a ClickHouse view filtered on unprotected over the outage window.

| Option | Verdict | Reasoning |
|---|---|---|
| Closed by default, open only for declared efficiency classes | Chosen | Safety where it matters, availability where it is cheap, and a record of every compromise. |
| Closed for everything | Deferred | The MVP posture. Moves to the chosen option in Phase 3 once the reconciliation report exists. |
| Open for everything during outages | Rejected | Converts a quorum outage into a correctness incident across every class at once. |

**What it buys**

- Quorum loss is an availability incident, not a correctness incident
- Efficiency workloads keep moving
- Every compromise is a bounded, queryable list

**What it costs**

- Correctness workloads stop during quorum loss
- Fail-open adds a local spool and a reconciliation path to build and test
- Owners must classify honestly; a misdeclared efficiency class is a silent correctness risk

**Choose differently when.** Flip a class to closed the first time anyone has to clean up after one of its unprotected grants. That cleanup is evidence the class was never an efficiency class.

**Why it holds up over time.** Safety-before-liveness is the one posture that never needs revisiting when the business grows; the only thing that changes is which classes deserve the exception, and that list lives in reviewed files.

> **Lesson.** An availability exception is acceptable when it is declared in advance, bounded, and leaves a list behind.

### Consensus and topology

*The store, where it runs, how many of it there are, and why none of it spans regions.*

#### ADR-05 · etcd as the lock log

**Status:** Accepted  ·  **Shown on views:** 02, 07, 14

*Which replicated, linearizable store holds lease state and mints the ordering?*

**Context.** The store needs linearizable conditional multi-key writes, leases whose expiry is a committed entry measured on the quorum's monotonic clock, a strictly increasing commit index, and a watch that resumes from a position. It has to be open source, run on the operator's hardware, and be operable by a platform team that already runs Kubernetes.

**Decision.** Use etcd on the 3.6 release line, run as dedicated clusters for lock traffic. The arbiter uses Txn for grants and releases, LeaseGrant and LeaseKeepAlive for sessions, header revisions for tokens and Watch for queues and audit. Lease checkpointing is enabled so a leader change carries remaining TTL forward.

**How it works on-premise.** Members run as a StatefulSet pinned to tainted nodes with local NVMe persistent volumes, guaranteed QoS and CPU pinning. --quota-backend-bytes is 8 GB, auto-compaction is periodic at one hour, defragmentation runs one member at a time off-peak. Client and peer TLS use certificates from a private CA; RBAC is enabled with per-prefix roles.

| Option | Verdict | Reasoning |
|---|---|---|
| etcd | Chosen | Every required primitive is native, and its Raft implementation is exercised by every Kubernetes cluster in existence. |
| Apache ZooKeeper | Rejected | Proven for exactly this job, with ephemeral sequential nodes and zxids. Loses on operational familiarity for a Kubernetes-native team and JVM tuning on the latency tail. |
| Consul | Rejected | Sessions and KV locks exist, but its lock-delay and session-invalidation semantics are coarser, and it brings a service-discovery product the estate does not need here. |
| Custom Raft on hashicorp/raft or etcd-io/raft | Deferred | Could remove the key-value layer's overhead at very high scale. Revisit only if a benchmarked shard cannot meet the per-cluster ceiling. |
| PostgreSQL advisory locks or SELECT FOR UPDATE | Right elsewhere | Right when the guarded resource is itself one PostgreSQL database; the lock and the write then share a transaction and no service is needed. |
| Redis with Redlock | Rejected | Relies on bounded clock drift and process pauses for safety and has no committed expiry. The failure this service exists for is the one it does not handle. |

**What it buys**

- No new consensus implementation to trust
- Large pool of engineers who already operate etcd
- Transactions, leases and watches in one API

**What it costs**

- Every key carries etcd's MVCC overhead and compaction obligations
- Write throughput per cluster is bounded, so sharding comes earlier than with a purpose-built log
- etcd's keepalive path is leader-served rather than logged, which the design accounts for in ADR-10

**Choose differently when.** Choose ZooKeeper where a team already runs it well for Kafka or HBase and has no Kubernetes estate. Choose a purpose-built Raft log if sustained lock writes exceed what ten entity shards can carry.

**Why it holds up over time.** etcd is a CNCF-graduated project that Kubernetes cannot exist without, which makes it one of the safest open-source bets for continued maintenance. The arbiter uses four etcd capabilities behind an internal interface, so a replacement is an arbiter change, not a protocol change.

> **Lesson.** Pick the consensus store your platform already depends on for survival; its maintenance is then someone else's existential problem.

#### ADR-06 · Dedicated etcd clusters on dedicated nodes, never the Kubernetes control plane's

**Status:** Accepted  ·  **Shown on views:** 07, 19

*Where do the etcd members physically run?*

**Context.** Grant latency is the fsync latency of the slowest member in the acknowledging majority. The Kubernetes control plane already runs an etcd, and reusing it is tempting. That etcd is sized and tuned for the API server, and a lock storm there degrades the cluster the lock service itself is deployed on.

**Decision.** Lock clusters are separate etcd clusters on dedicated worker nodes with local NVMe, one member per hall for the entity tier and a 2-2-1 split for coord. Disks are encrypted with LUKS and unlock through Clevis against Tang servers on a separate network.

**How it works on-premise.** Node labels lock-etcd=true and a NoSchedule taint; local persistent volumes through the local static provisioner; WAL and data on the same NVMe with nothing else on it; ionice and CPU manager static policy. Tang servers run in two halls; a node that cannot reach Tang at boot stays locked rather than falling back to a passphrase.

| Option | Verdict | Reasoning |
|---|---|---|
| Dedicated clusters, dedicated nodes, local NVMe | Chosen | Predictable tail latency and no shared fate with the orchestrator. |
| Reuse the Kubernetes control-plane etcd | Rejected | Couples lock load to cluster health, and the API server's own leases would compete with lock traffic. |
| etcd on shared workers with network block storage | Rejected | Network storage fsync tails of tens of milliseconds consume the whole latency budget. |
| etcd on bare metal outside Kubernetes | Right elsewhere | Equally valid where the platform team runs systemd services well; loses the uniform deployment and rollout tooling. |

**What it buys**

- Stable fsync latency and an isolated failure domain
- A stolen disk reveals nothing; a reboot needs no human
- Failure domains are explicit in the node placement

**What it costs**

- Dedicated hardware that sits well below capacity by design
- Tang becomes a boot-time dependency to operate
- Hall independence must be verified, not assumed

**Choose differently when.** Run on shared workers only in a non-production environment, or where measured fsync p99 on shared storage stays under 2 ms at burst.

**Why it holds up over time.** Separating a lock quorum from the orchestrator's own state is a boundary that stays correct however the orchestrator evolves, and local-disk consensus remains the lowest-latency option on any hardware generation.

> **Lesson.** A consensus system's latency is its slowest disk in the majority; buy the disk before tuning the software.

#### ADR-07 · Two cluster tiers, coord and entity, chosen by the lock class

**Status:** Accepted  ·  **Shown on views:** 09, 22

*One service for coarse and fine-grained locking, or two?*

**Context.** Leader election needs tens of long-held, highly available locks. Entity locking needs hundreds of thousands of short, high-churn ones. On one cluster, entity churn inflates compaction, defragmentation and tail latency for the leader-election locks the rest of the platform depends on. Two clusters double the operational surface and require a routing rule that callers will eventually get wrong.

**Decision.** Same protocol, same client library, two tiers. The coord tier is a five-member cluster tuned for availability; the entity tier is sharded three-member clusters tuned for write throughput. The tier is a required field on the lock class, so routing is decided once by the owner in review and never by a caller.

**How it works on-premise.** The arbiter holds one clientv3 client per cluster and selects by class tier and, for entity, by the shard map (ADR-09). The coord cluster also stores epoch allocation and the shard map itself. In the MVP both tiers point at one cluster; the split is a configuration change once the coord cluster exists.

| Option | Verdict | Reasoning |
|---|---|---|
| Two tiers behind one protocol, tier on the class | Chosen | Blast radius isolation without asking callers to route. |
| One cluster for everything | Deferred | Acceptable for the MVP's volume; split at Phase 2 or on the first latency incident traced to entity churn. |
| Two separate services with separate APIs | Rejected | Two libraries and two sets of semantics for what is one primitive. |

**What it buys**

- Leader election survives an entity-tier incident
- Each tier tuned for its own shape
- No caller-side routing mistakes

**What it costs**

- More clusters to operate, patch and back up
- Moving a class between tiers is a rehome, not an edit

**Choose differently when.** Stay on one tier if the estate never exceeds a few thousand concurrent entity locks; the operational cost of a second cluster then buys nothing measurable.

**Why it holds up over time.** The tier is a property of a class, not of the infrastructure, so new tiers, such as a regional or a global coord, slot in without any existing class or caller changing.

> **Lesson.** Isolate by workload shape, and put the routing decision where it is made once, in review.

#### ADR-08 · Regional quorums only; no stretched quorum until a global resource is named and priced

**Status:** Accepted  ·  **Shown on views:** 22

*What is the quorum topology across regions?*

**Context.** A stretched quorum coordinates globally and pays inter-region latency on every acquisition, including the local majority. Regional clusters are fast and cannot coordinate a globally unique action. A hierarchy, regional clusters holding leases on a global lock, adds a second lease layer with its own expiry semantics and failure modes.

**Decision.** Each region runs its own coord and entity clusters, and no lock spans regions. A key's tenant namespace is homed to one region. A global lock is not offered until a named resource genuinely needs one, and then only as a separate stretched coord cluster with its acquisition latency, p99 around 150 ms, published before anyone commits to it.

**How it works on-premise.** Region is a field on the namespace. The arbiter in region A refuses keys homed to region B with a redirect error naming the region, rather than forwarding, so cross-region latency is never hidden inside a call.

| Option | Verdict | Reasoning |
|---|---|---|
| Regional only, global deferred and priced | Chosen | Every local lock stays fast, and the expensive option is a decision rather than a default. |
| Stretched quorum across three regions | Rejected | Every acquisition pays a cross-region round trip for the benefit of a small minority of locks. |
| Hierarchical regional and global leases | Rejected | Two expiry layers whose interaction under partition is harder to reason about than either alone. |

**What it buys**

- Local latency for every lock
- A region's isolation cannot stall another region's locks

**What it costs**

- No globally unique action without a separate, slower cluster
- Namespaces must be homed deliberately

**Choose differently when.** Build the stretched coord cluster when a named resource, such as a global ledger migration or a cross-region singleton, cannot be partitioned by region and its owner accepts the published latency.

**Why it holds up over time.** Physics does not improve. Inter-region latency will be roughly what it is now in ten years, so a topology that keeps it out of the common path does not age.

> **Lesson.** Price the global option before anyone asks for it, and make them ask.

#### ADR-09 · Home entity keys by rendezvous hashing, and fence every move with an epoch advance

**Status:** Accepted  ·  **Shown on views:** 09, 22

*How is the key space spread across entity shards, and how do keys move without token regression?*

**Context.** A key must be deterministically homed to exactly one cluster. Adding a shard must move few keys. Tokens are revision-based and each cluster has its own revision space, so a key moved to a cluster with a lower revision would issue lower tokens than its old home did.

**Decision.** Entity keys are homed by rendezvous hashing over namespace and path prefix against a versioned shard map stored in the coord cluster. Adding a shard moves only keys whose highest score changes to it. Before the destination accepts moved keys, its epoch is advanced above the source's epoch, and the move proceeds only for ranges with no current holders.

**How it works on-premise.** The shard map is a versioned document in /sys on the coord cluster, watched by every arbiter. A rebalance controller drains a range by refusing new grants for it, waits for holds to release or expire, advances the destination epoch, flips the map version and resumes.

| Option | Verdict | Reasoning |
|---|---|---|
| Rendezvous hashing with epoch-fenced moves | Chosen | Minimal movement, no ring tokens to manage, and the same epoch mechanism as restore. |
| Consistent-hash ring with virtual nodes | Rejected | Works, with more state to balance and no advantage at tens of shards. |
| Manual range assignment by namespace | Right elsewhere | Fine for a handful of very large tenants who want their own shard; supported as an override in the map. |

**What it buys**

- Adding a shard moves about 1/N of keys
- One mechanism for restore and rehome safety

**What it costs**

- A moving range is briefly unavailable for new grants
- The shard map is one more critical document on the coord cluster

**Choose differently when.** Prefer explicit per-tenant assignment when the estate is a small number of very large tenants whose isolation matters more than balance.

**Why it holds up over time.** Rendezvous hashing has no tuning parameters that drift with scale, and epoch-fenced moves make every future rebalance safe by the same rule that makes a restore safe.

> **Lesson.** When moving state can break an ordering, reuse the mechanism that already protects the ordering, rather than inventing a second one.

### Leases and holders

*Sessions, TTLs and what a holder does in the dark.*

#### ADR-10 · A session is one etcd lease per identity and TTL; every hold attaches to it

**Status:** Accepted  ·  **Shown on views:** 11, 14

*How is renewal traffic kept from dominating the write path?*

**Context.** 250,000 held leases at a 15 s TTL renewed every 5 s is 50,000 renewals a second. Unbatched, that is four times the steady write target and more than any single etcd cluster should carry. The requirement asks for batched session renewal as a requirement, not an optimisation.

**Decision.** The client library opens one etcd lease per distinct (SPIFFE identity, TTL) pair and attaches every hold of that TTL to it. One LeaseKeepAlive renews all of them. A session's expiry revokes every attached hold in one committed entry. Renewal and expiry of individual holds do not exist as separate operations.

**How it works on-premise.** At 40,000 sessions with 5 s keepalives, renewal is 8,000 keepalives a second, handled by the etcd leader without a Raft proposal; lease checkpoints are proposed periodically so remaining TTL survives a leader change. The pinned etcd release must confirm leadership before honouring a keepalive, and the release gate injects leader isolation to verify that no deposed leader extends a lease.

| Option | Verdict | Reasoning |
|---|---|---|
| One lease per session, holds attached | Chosen | Batching falls out of the data model instead of being built as a feature. |
| One etcd lease per lock | Rejected | Simple, and renewal cost scales with held locks rather than with processes. |
| Heartbeat keys written by the application | Rejected | Every heartbeat is a consensus write, and expiry becomes a timer decision on one node. |

**What it buys**

- Renewal cost scales with processes, not locks
- A crashed process releases everything in one entry
- Batched renewal of 500 holds costs the same as one

**What it costs**

- A hold cannot have its own TTL different from its session's; a class with a different TTL gets its own session
- Keepalives are not log entries, which departs from the requirement's wording and is covered by leadership confirmation and checkpoints instead

**Choose differently when.** Use per-lock leases when a caller holds one or two locks per process and needs different expiry per lock; the batching benefit is then negligible.

**Why it holds up over time.** Tying liveness to the process rather than to each resource it touches is how every long-lived coordination system has ended up, from Chubby sessions to ZooKeeper sessions. It will outlast the store.

> **Lesson.** Renew the thing that can die, the process, not every thing it happens to hold.

#### ADR-11 · TTL is chosen per class within a platform range, with its renewal cost printed in review

**Status:** Accepted  ·  **Shown on views:** 04, 23

*How long is the lease, and who chooses it?*

**Context.** A short TTL recovers faster from a dead holder but multiplies keepalive load and turns latency spikes into false expiries. A long TTL is cheap and leaves a crashed holder blocking progress for its whole duration. Whether a false expiry is harmless depends on whether the resource is fenced, so this decision depends on ADR-01.

**Decision.** The platform allows 5 s to 300 s with a 15 s default. Each class chooses within that range; callers cannot override per request. Fenced classes may go short, because a false expiry costs only a retry. Advisory classes have a floor of 30 s, because a false expiry there can produce a duplicate write. The class pull request prints keepalive load and liveness bound for the chosen value.

**How it works on-premise.** A CI check on the class repository computes expected sessions × keepalives per second from the namespace's current session count in Prometheus and comments on the pull request. The admission webhook enforces the range and the advisory floor.

| Option | Verdict | Reasoning |
|---|---|---|
| Per class, within a range, cost printed | Chosen | The owner decides with the numbers in front of them, once. |
| One platform-wide TTL | Rejected | Leader election and 2-second entity writes want different answers. |
| Caller chooses per acquisition | Rejected | Unreviewed, uncosted, and indistinguishable from a bug when a caller passes 300 s. |

**What it buys**

- Cost and liveness trade-off is explicit and reviewed
- False-expiry risk is tied to enforcement posture

**What it costs**

- Owners must understand the trade-off
- Different TTLs in one process mean more sessions

**Choose differently when.** Allow per-request TTL within the class range if a class guards work units with highly variable duration and fencing is in place, so a shorter request cannot cause damage.

**Why it holds up over time.** TTL is the primary cost lever of any lease system. Keeping it as a reviewed, costed configuration value means it can be retuned as hardware and networks change without any code change.

> **Lesson.** Put the cost of a knob next to the knob.

#### ADR-12 · Stop starting work at the first failed keepalive; fence at a deadline measured from send time

**Status:** Accepted  ·  **Shown on views:** 18, 15

*What does a holder do when it cannot renew but has not been told it lost the lock?*

**Context.** Aborting at the first failure is safe and wastes work on every network blip. Continuing to a locally computed deadline trusts local timing, which a stop-the-world pause violates. Continuing until told is unsafe by construction, because a partitioned holder is never told.

**Decision.** Two stages in the client library. At the first failed keepalive, stop granting permission to start new units of work and let in-flight work finish. At the local deadline, call onFence. The deadline is the monotonic send time of the last successful keepalive plus the TTL minus a one-second margin. Using send time guarantees the local deadline precedes the leader's expiry for that keepalive.

**How it works on-premise.** The Go library uses time.Now() monotonic readings; Java uses System.nanoTime(); Python uses time.monotonic(). The application checks lock.canStart() before each unit of work, and onFence is a required constructor argument. The library also exposes remainingTTL() so a caller can decline to start a unit it cannot finish.

| Option | Verdict | Reasoning |
|---|---|---|
| Two-stage drain then fence at a send-time deadline | Chosen | Cheap under blips, conservative under partitions, and honest about pauses. |
| Abort immediately on first failure | Rejected | Safe, and turns every 200 ms blip into aborted work across the fleet. |
| Continue until the service says the lock is lost | Rejected | A partitioned holder is never told; the requirement names this as unsafe. |

**What it buys**

- Holder always believes it has lost before the log says so
- Transient failures cost a pause, not a redo

**What it costs**

- Applications must structure work into units that can check canStart()
- Does nothing for a pause that swallows the deadline, which only the token handles

**Choose differently when.** Abort immediately where units of work are cheap to redo and the resource is advisory; the conservative path then costs little.

**Why it holds up over time.** Monotonic clocks, send-time deadlines and a callback are available in every language runtime and will stay so. The rule is independent of the store.

> **Lesson.** Measure your deadline from the last moment you know was true, which is when you asked, not when you heard back.

#### ADR-13 · Re-entrancy is counted in the client library; the log sees only the outermost acquire and release

**Status:** Accepted  ·  **Shown on views:** 06

*Where is the re-entrant hold count kept?*

**Context.** The requirement asks for re-entrant acquisition by the same session with a hold count, released at zero, and refusal when the same identity holds under a different session. Counting in the log costs a consensus write per nested acquire.

**Decision.** The library keeps the count per session and key and only calls the service for the outermost acquire and the final release. The arbiter refuses a grant when the key is held by the same SPIFFE identity under a different session lease, returning a leak error rather than a contention error.

**How it works on-premise.** The holder key's value carries the SPIFFE ID and session lease ID. The arbiter's grant transaction reads the existing value on failure and classifies the refusal. Re-entrancy is available from Phase 2.

| Option | Verdict | Reasoning |
|---|---|---|
| Count in the library, leak check in the arbiter | Chosen | Zero consensus cost for nesting; the dangerous case is still caught server-side. |
| Count in the holder key | Rejected | Correct and costs two writes per nested call for no safety gain within one session. |

**What it buys**

- Nested acquisition is free
- Leaks across sessions are named as leaks

**What it costs**

- The count is lost if the library crashes, which is also when the session and all its holds end, so nothing is left inconsistent

**Choose differently when.** Move the count server-side if multiple processes ever share one session, which this design forbids.

**Why it holds up over time.** Keeping per-process bookkeeping in the process is the arrangement that stays cheap as call patterns change.

> **Lesson.** Only send the log facts that another process needs to know.

### Contention

*Queues, fairness and why a notification never grants anything.*

#### ADR-14 · In-memory FIFO on a key-affine arbiter by default; durable waiter keys as a class option

**Status:** Accepted  ·  **Shown on views:** 16, 10

*Is the wait queue durable state?*

**Context.** Replicating the queue through consensus preserves arrival order across failures and puts every enqueue and cancellation on the write path, which for a hot key can exceed its grant traffic. In-memory queueing is nearly free and loses order when the queue's host fails.

**Decision.** Envoy ring-hashes on the lock key so every waiter for a key lands on the same arbiter, which queues them in arrival order in memory. Order is declared best-effort: it survives an etcd leader change and is lost on arbiter restart or ring rebalance, after which waiters retry. A class may set fairness: durable, which stores waiters as keys under /q ordered by create revision, at two consensus writes per waiter, printed on the class PR.

**How it works on-premise.** Envoy route uses hash_policy on the x-lock-key header with ring_hash load balancing across arbiter endpoints. Durable waiters use the etcd concurrency recipe: put /q/<key>/<leaseID> on the waiter's session lease and grant when its create revision is the lowest.

| Option | Verdict | Reasoning |
|---|---|---|
| In-memory by default, durable per class | Chosen | Most classes get fairness for free; the few that need it pay for it knowingly. |
| Durable queue for every class | Rejected | Doubles write load on hot keys to protect order that most callers do not rely on. |
| No queue; callers poll with backoff | Rejected | Starvation under contention and a thundering herd on every release. |

**What it buys**

- No consensus cost for queueing on most keys
- One release produces one grant attempt

**What it costs**

- A waiter can lose its place on arbiter restart
- Key-affine routing means one arbiter's loss affects all waiters for its keys

**Choose differently when.** Make durable the default if contention reports show waiters routinely losing position during arbiter rollouts on correctness classes.

**Why it holds up over time.** The choice is a class field, so the default can move in either direction as hardware makes consensus writes cheaper, without any caller changing.

> **Lesson.** Make fairness a priced option, not a universal tax.

#### ADR-15 · A watch notification triggers an attempt, never a grant

**Status:** Accepted  ·  **Shown on views:** 16

*What may a release notification cause?*

**Context.** Watch streams can drop, reorder around reconnects and lag. A design that treats a delete event as permission to hold turns a lost or late event into a double grant.

**Decision.** Every grant, including one triggered by a notification, is a full grant transaction comparing key absence and epoch. A waiter that misses a notification falls back to polling every second with jitter until its deadline. The watch stream is authoritative for nothing.

**How it works on-premise.** The arbiter watches the key from the revision returned by the failed grant, so no event between the check and the watch is missed. On watch cancellation due to compaction, it retries the grant transaction immediately and re-watches from the new revision.

| Option | Verdict | Reasoning |
|---|---|---|
| Notification as a hint, transaction as the grant | Chosen | A lost event costs a second of delay, never correctness. |
| Grant directly on delete event | Rejected | Saves one round trip and makes safety depend on stream delivery. |

**What it buys**

- Watch failures are latency incidents only
- No special-case grant path to test

**What it costs**

- One extra transaction per contended release

**Choose differently when.** None within this design. A notification that grants is a different, weaker service.

**Why it holds up over time.** Separating hints from authority is independent of any watch implementation and survives every change to the notification transport.

> **Lesson.** Derived streams may wake you; only the log may let you in.

### Enforcement at the resource

*Where two writers are actually kept apart, and what happens where they are not.*

#### ADR-16 · The resource's own conditional write holds the high-water mark; validation API is the fallback

**Status:** Accepted  ·  **Shown on views:** 14, 08

*Who holds the fencing high-water mark?*

**Context.** Resource-side marks are cheapest and fastest but require every resource to implement the rule. A validation call back into the lock service centralises the rule and puts the service on every write path. Using the store's own conditional write is often strictly better, and raises the question of whether the lock was needed at all.

**Decision.** The pattern is the store's own atomic conditional write: UPDATE ... SET fence = $token WHERE id = $id AND fence < $token for relational stores, and conditional put with an ETag or version precondition for object stores. The ValidateToken API exists from Phase 2 only for resources that genuinely cannot store a mark, is rate-limited separately, and is reported per namespace as a dependency.

**How it works on-premise.** Reference implementations ship for PostgreSQL (a fence column updated in the same statement as the guarded write) and for S3-compatible stores including MinIO (version metadata checked with If-Match). ValidateToken is a linearizable read of the holder key comparing the presented token with the current grant.

| Option | Verdict | Reasoning |
|---|---|---|
| Store's conditional write as the fence | Chosen | Atomic with the write, no extra hop, no new dependency on the lock service. |
| Validation call on every write | Deferred | Phase 2 fallback. Not atomic with the write unless the resource holds its own lock between check and write. |
| Sidecar proxy that fences writes | Rejected | Protocol-specific per store and adds a component on every data path. |

**What it buys**

- Mutual exclusion enforced where the write lands, atomically
- The lock service stays off data paths

**What it costs**

- Each resource type needs its own implementation
- Owners discover some stores cannot do it, which feeds the advisory register

**Choose differently when.** Prefer the validation API for a resource type with low write volume, many instances and no conditional write, where one central rule is worth the hop.

**Why it holds up over time.** Conditional writes are a primitive of every serious store and are only becoming more common. A fence implemented as one outlives both the lock service and the store's current version.

> **Lesson.** If the store can compare-and-set, the fence is one clause, and it is worth asking whether you needed the lock.

#### ADR-17 · Ship a reference fence and a pause test kit; declare fenced only on a passing test

**Status:** Accepted  ·  **Shown on views:** 04

*How does a resource owner prove their resource is fenced?*

**Context.** Owners will implement the fence subtly wrong: a check in one statement and the write in another, a less-than-or-equal instead of less-than, a mark per table instead of per record. A class marked fenced on assertion gives false confidence in exactly the case the platform claims to protect.

**Decision.** The platform ships the reference fence as copyable code and a test kit that acquires a lock, writes, freezes the holder with SIGSTOP past its TTL, lets a second holder acquire and write, resumes the first and asserts its write is rejected. A class can be marked fenced only when a result object for that kit run exists.

**How it works on-premise.** The kit is a container image run as a Kubernetes Job against the owner's staging resource. It writes a PauseTestResult custom resource that the class controller checks. Results expire after a year or on any change to the resource's schema version label.

| Option | Verdict | Reasoning |
|---|---|---|
| Reference code plus mandatory pause test | Chosen | The claim of fenced is backed by an observed rejection. |
| Documentation and code review only | Rejected | Reviewers miss non-atomic checks. |
| Platform implements fences for every store | Rejected | Unbounded scope and ownership of other teams' data paths. |

**What it buys**

- Fenced means tested
- The kit doubles as onboarding documentation

**What it costs**

- A staging resource is required
- Test results need renewal

**Choose differently when.** Relax to annual attestation for resources whose write path has not changed in a year and whose owners have passed the test before.

**Why it holds up over time.** A behavioural test survives rewrites of the resource, changes of language and changes of team. It checks the property, not the code that once had it.

> **Lesson.** Declare a safety property only after you have watched it refuse something.

### Access and identity

*Why there is a service at all, who may lock what, and who may break a lock.*

#### ADR-18 · A stateless Go arbiter behind Envoy, rather than a thick client straight to etcd

**Status:** Accepted  ·  **Shown on views:** 02, 07, 08

*Why run a service at all, rather than a client library that talks to etcd?*

**Context.** A thick client is fewer moving parts and no extra hop. It also means 2,000 workloads hold etcd credentials, 40,000 sessions hold etcd connections, class rules and quotas are enforced in every language separately, and an etcd change requires every team to upgrade a library.

**Decision.** Callers speak lock.v1 gRPC through Envoy to a stateless Go arbiter, which is the only component with an etcd write credential for grants. The arbiter enforces epoch, class, authorisation, quota and queueing, and is stateless for correctness. Envoy terminates SPIFFE mTLS, applies per-identity local rate limits and routes by key.

**How it works on-premise.** Envoy runs as a Deployment of six replicas with SDS from the SPIRE agent. The arbiter is built with Go and etcd's clientv3, one client per cluster, nine replicas spread across halls. The proto is managed with buf, and breaking-change checks run on every pull request.

| Option | Verdict | Reasoning |
|---|---|---|
| Arbiter service behind Envoy | Chosen | One enforcement point, one etcd credential holder, and libraries that stay thin. |
| Thick client straight to etcd | Rejected | Fine for one team; for 300 namespaces it scatters credentials and rules across every language. |
| Arbiter in Java or Rust | Rejected | Both capable. Go matches etcd's own client and tooling, which matters most when debugging the client under faults. |
| HAProxy or NGINX as fan-in | Right elsewhere | Good gRPC proxies; Envoy wins on SPIFFE-native SDS and hash policy on a header. |

**What it buys**

- Rules enforced once
- etcd protected from connection count
- Libraries can be thin in every language

**What it costs**

- An extra hop of about 1 to 3 ms
- Two more tiers to deploy and monitor

**Choose differently when.** Use a thick client for a single-team platform with one language and a handful of services, where the arbiter's enforcement buys little.

**Why it holds up over time.** The gRPC contract is the stable boundary; everything behind it, including the store, can be replaced without callers upgrading. That is the property that keeps a shared service alive for a decade.

> **Lesson.** Put the contract where you want freedom to change what is behind it.

#### ADR-19 · SPIFFE workload identity, and authorisation as prefix grants rather than a policy engine

**Status:** Accepted  ·  **Shown on views:** 24

*How is a workload identified, and how is it limited to its own keys?*

**Context.** Every session and hold must be bound to an authenticated identity, and a workload may lock only keys it owns. Namespaces must not leak existence to one another through errors or timing. The authorisation question is narrow: may this identity lock under this prefix.

**Decision.** Workloads authenticate with X.509 SVIDs from SPIRE over mTLS. Authorisation is a list of prefix grants per SPIFFE ID, declared in the class registry and evaluated in-process by the arbiter as a longest-prefix match before any etcd read. Denials return one error, in constant time, whether or not the key exists.

**How it works on-premise.** SPIRE server runs HA with its datastore on PostgreSQL; agents run as a DaemonSet with Kubernetes workload attestation. SVIDs live one hour. Prefix grants are compiled into a radix tree on each class change.

| Option | Verdict | Reasoning |
|---|---|---|
| SPIRE plus in-process prefix grants | Chosen | Attested identity and an authorisation check measured in microseconds. |
| Open Policy Agent sidecar | Rejected | A general engine for a question that is a prefix match, on a path with a 40 ms p99. |
| Kubernetes service account tokens | Rejected | Ties identity to one cluster and to bearer tokens rather than attested certificates. |

**What it buys**

- Identity is attested and short-lived
- Authorisation adds no network hop

**What it costs**

- SPIRE is a platform dependency to operate
- Complex conditions, such as time-of-day rules, are not expressible, deliberately

**Choose differently when.** Add a policy engine if authorisation needs attributes beyond identity and prefix, for example data classification of the guarded resource.

**Why it holds up over time.** SPIFFE is an open standard with multiple implementations, so identity survives a change of SPIRE, of orchestrator or of cloud. Prefix grants are data, portable to any future engine.

> **Lesson.** Match the authorisation mechanism to the shape of the question, not to the most general tool available.

#### ADR-20 · Force-release requires two operators, a reason, and a delete-only etcd role

**Status:** Accepted  ·  **Shown on views:** 05, 25

*How can anyone other than the holder end a hold without making things worse?*

**Context.** Leaked locks, holders alive and renewing but never releasing, are not automatically resolvable. Operators need a way to clear them. On an unfenced resource a force-release can let two writers in, and the operator must be told this before acting, not afterwards.

**Decision.** Force-release is available only through the Admin API, requires a requester and a different approver, both signed in through Keycloak with MFA and the lock-operator role, and a stated reason. The Admin API shows the class's fenced status before approval. The release is one transaction that deletes the hold and writes a /sys/force record, and may optionally revoke the holder's whole session.

**How it works on-premise.** Keycloak realm with a lock-operator role and required OTP or WebAuthn. The Admin API's etcd role permits delete on /l and put on /sys/force only. Force records carry a 24-hour lease; the tailer copies them to ClickHouse and the seven-year archive before they expire.

| Option | Verdict | Reasoning |
|---|---|---|
| Two-person, reasoned, delete-only credential | Chosen | Slow enough to prevent reflex, fast enough for an incident. |
| Single operator with audit | Rejected | Audit explains the double write afterwards; it does not prevent it. |
| etcdctl access for SREs | Rejected | Unbounded write access to the one store where a mistake is a duplicate grant. |

**What it buys**

- Every force-release carries who, why and whether it was safe
- No human holds a general etcd write credential

**What it costs**

- Needs a reachable second approver at night
- Keycloak outage blocks force-release, by design

**Choose differently when.** Allow a single-operator break-glass path with post-hoc review only if two-person approval has demonstrably extended incidents, and only for fenced classes.

**Why it holds up over time.** Two-person control with a recorded reason is how high-consequence operations have been governed for decades, and it survives any change in the tools that implement it.

> **Lesson.** Tell operators whether an action is safe at the moment they choose it.

### Evidence and operations

*Audit, telemetry, release gating and configuration.*

#### ADR-21 · Audit is tailed from etcd revisions into ClickHouse, never dual-written by the arbiter

**Status:** Accepted  ·  **Shown on views:** 12, 10

*How is grant, release, expiry and force-release history captured completely?*

**Context.** An arbiter writing events alongside its grants has a dual-write problem: a crash between the two leaves a grant with no event. Worse, expiries are committed by etcd itself when a session lapses; no application code sees them, and they are the events investigators most need.

**Decision.** An audit tailer per cluster watches the whole lock key space from its last checkpointed revision and inserts events into ClickHouse keyed by (cluster ID, revision). Grant history is exported to Parquet on MinIO for a year; force-release records go to a bucket with object lock for seven years. The tailer alarms at half of etcd's compaction window and records any gap explicitly.

**How it works on-premise.** ClickHouse runs replicated with ReplacingMergeTree on (cluster_id, revision), making replays idempotent. The tailer is a singleton per cluster through an election on the coord cluster and checkpoints every few seconds. MinIO runs erasure-coded across halls with object lock in retention mode.

| Option | Verdict | Reasoning |
|---|---|---|
| Revision tailer into ClickHouse and MinIO | Chosen | Complete by construction, including expiries, and idempotent. |
| Arbiter publishes events to Kafka | Rejected | Misses expiries, dual-writes grants, and adds a broker the platform does not otherwise need here. |
| PostgreSQL as audit store | Right elsewhere | Fine up to tens of millions of events; ClickHouse is chosen for scans over 90 days of several million events a day. |

**What it buys**

- Every committed change is captured, including expiries
- No broker on the grant path
- Cost and contention reports come from the same data

**What it costs**

- Tailer lag must stay well inside the compaction window
- ClickHouse is another stateful system to run

**Choose differently when.** Emit from the arbiter as well if audit must carry context the log does not have, such as denial reasons; keep the tailer as the source of truth for grants.

**Why it holds up over time.** A change-data-capture pattern keyed on the log's own position is independent of both the audit store and the arbiter, and survives replacing either.

> **Lesson.** When the log is the truth, read history from the log rather than asking every writer to remember to report it.

#### ADR-22 · Metrics by namespace and class only; tokens never in telemetry

**Status:** Accepted  ·  **Shown on views:** 21

*What does the service emit, and with which labels?*

**Context.** Operators need grant latency, contention ratio, hold times, expiry without release and renewal failures per namespace and class. Four million keys a day as a metric label would overwhelm any time-series store. Tokens are capabilities, and a token in a trace or log is a token anyone with log access can present.

**Decision.** Metrics use namespace, class, tier and cluster labels only. Per-key questions go to the inspection API. The client libraries never record the token in spans, logs or errors; the OTel Collector runs a redaction processor on lock.token as defence in depth. Expiry without release and longest hold are routed to owning teams as client-health signals.

**How it works on-premise.** Prometheus scrapes arbiters, Envoy and etcd; Grafana holds SLO dashboards; Alertmanager routes by namespace owner label. OTel Collector receives traces and logs and forwards to Tempo and Loki. A CI lint fails any metric definition that adds a key label.

| Option | Verdict | Reasoning |
|---|---|---|
| Bounded labels plus inspection API | Chosen | Time series stay cheap; per-key answers stay exact. |
| Per-key metrics | Rejected | Cardinality explosion in the first week. |
| Tokens in logs for debugging | Rejected | Turns log access into write authority at guarded resources. |

**What it buys**

- Stable Prometheus cost as key volume grows
- No capability leaks through telemetry

**What it costs**

- Debugging one key requires the inspection API rather than a dashboard

**Choose differently when.** Add exemplars carrying key hashes if per-key latency investigation becomes routine; never raw keys or tokens.

**Why it holds up over time.** Bounded label discipline is what keeps any metrics backend affordable; OpenTelemetry keeps the emitting side portable across backends.

> **Lesson.** A label is a cost multiplier, and a secret in telemetry is a secret published.

#### ADR-23 · No release without a fault-injected history checked for double grants

**Status:** Accepted  ·  **Shown on views:** 20

*How is the zero-double-grant property protected against future changes?*

**Context.** The property fails only under timing the unit tests never produce: partitions mid-transaction, a deposed leader accepting a keepalive, a pause at the wrong moment. The engineers who understand the design will not be the ones making changes in five years. etcd upgrades can change lease behaviour without any change in this codebase.

**Decision.** Every arbiter release and every etcd version change runs a suite on real clusters under Chaos Mesh faults: partitions, member kills, leader isolation, clock steps, disk latency and process pauses. Every client operation is recorded with start and end times, and Porcupine checks each history against a sequential lock model. One violation blocks the release and attaches the history.

**How it works on-premise.** The suite runs in a dedicated staging namespace on the same node types as production. Gitea Actions builds and signs images with cosign into Harbor; Argo CD promotes only digests that carry a passing gate attestation. etcd versions are pinned by digest.

| Option | Verdict | Reasoning |
|---|---|---|
| Chaos Mesh plus Porcupine on every change | Chosen | Checks the property itself on real software under the faults that break it. |
| Periodic external Jepsen analysis | Right elsewhere | Valuable as an annual deep review; too slow and costly to gate every release. |
| Unit and integration tests only | Rejected | Cannot produce the interleavings that matter. |

**What it buys**

- A regression in the core property cannot ship silently
- etcd upgrades are verified, not trusted

**What it costs**

- Release cycle lengthened by hours
- A realistic staging environment to maintain
- Checker model must be updated with the protocol

**Choose differently when.** None for correctness-class releases. Documentation-only or dashboard changes skip the gate.

**Why it holds up over time.** The gate encodes the design's central claim as an executable check, so the claim survives staff turnover, rewrites and store replacement.

> **Lesson.** The only safety property you keep is the one a machine re-proves on every change.

#### ADR-24 · Lock classes and grants are reviewed CRDs delivered by Argo CD

**Status:** Accepted  ·  **Shown on views:** 04, 09

*Where do class definitions and prefix grants live, and who approves them?*

**Context.** Classes carry the decisions that make locks safe or unsafe: enforcement, posture, TTL and tier. They must be reviewed by the resource owner, versioned, diffable and reversible, and they must not depend on an API only the platform team can call.

**Decision.** LockClass and LockGrant custom resources live in a Gitea repository with CODEOWNERS per namespace. Argo CD syncs them to each cluster; a validating webhook enforces required fields, TTL range and posture rules; arbiters watch them and keep the last good set if the Kubernetes API is unavailable.

**How it works on-premise.** One repository per organisation with a directory per namespace. CI prints renewal cost and liveness bound for each change. Arbiters cache classes in memory and serve from the cache during API outages.

| Option | Verdict | Reasoning |
|---|---|---|
| Git-reviewed CRDs via Argo CD | Chosen | Review, history and rollback come from tools teams already use. |
| Admin API writes classes into etcd | Rejected | Loses review workflow and puts configuration in the lock log's blast radius. |
| ConfigMaps | Rejected | No schema validation or typed status. |

**What it buys**

- Every safety posture has an author, a reviewer and a date
- Configuration outlives the people who wrote it

**What it costs**

- Class changes take a pull request, not an API call
- Kubernetes API is a dependency for change, not for serving

**Choose differently when.** Offer an API for self-service efficiency classes if pull-request latency becomes a real adoption barrier; keep correctness classes in review.

**Why it holds up over time.** Declarative configuration in version control is the most portable form a decision can take; it survives a change of orchestrator, delivery tool or team.

> **Lesson.** Keep safety decisions where they are reviewed, versioned and attributable.

## Every package used, in one table

Every open-source package named on the views, what it is, and the job it does here.

| Package | What it is | What it does here | Considered instead |
|---|---|---|---|
| etcd | A distributed key-value store with Raft consensus, leases, transactions and watches | The lock log: holds leases, grants and epochs, and supplies the revision used in tokens | Apache ZooKeeper, Consul |
| Envoy | A layer-7 proxy with native gRPC and SDS support | Fan-in tier: mTLS termination, per-identity rate limits, key-affine routing | HAProxy, NGINX |
| SPIRE | The reference implementation of the SPIFFE workload identity standard | Issues one-hour X.509 SVIDs that bind sessions and holds to workloads | cert-manager with a private CA |
| Keycloak | An open-source identity and access management server | Operator sign-in with MFA and the lock-operator role for force-release | Dex |
| ClickHouse | A column-oriented analytical database | Audit store for 90 days of grant history, contention and cost reports | PostgreSQL, OpenSearch |
| MinIO | An S3-compatible object store | Snapshot vault, one-year Parquet archive and seven-year object-locked force-release archive | Ceph RADOS Gateway |
| Prometheus, Alertmanager, Grafana | Metrics collection, alert routing and dashboards | Lock SLIs, consensus health, holder health and paging | VictoriaMetrics |
| OpenTelemetry Collector, Tempo, Loki | Telemetry pipeline, trace store and log store | Traces and logs with token redaction | Jaeger, OpenSearch |
| Argo CD | A GitOps continuous delivery controller for Kubernetes | Syncs LockClass and LockGrant resources and service releases | Flux |
| Gitea and Gitea Actions | A self-hosted Git service with CI runners | Class repository, code review and builds | GitLab CE |
| Harbor and cosign | A container registry and a signing tool from Sigstore | Signed, admission-checked images | Distribution registry with Notation |
| Chaos Mesh | A Kubernetes-native fault injection platform | Partitions, pauses, clock steps and disk faults in the release gate | LitmusChaos |
| Porcupine | A fast linearizability checker written in Go | Checks recorded client histories for overlapping exclusive grants | Knossos, Elle (Jepsen) |
| buf | Protobuf tooling for linting and breaking-change detection | Keeps lock.v1 and admin.v1 compatible | protolock |
| LUKS, Clevis and Tang | Linux disk encryption with network-bound automatic unlock | Encrypts etcd disks at rest without manual unlock on reboot | Self-encrypting drives |