Architecture Views

Does the service promise that two processes never act at once, or only that grants are ordered?

Context

The requirement asks for correctness locks where two holders cause damage. But a holder paused past its lease cannot be stopped by any protocol, so exclusion can only be enforced where the write lands. Requiring every guarded resource to be fenced before a correctness lock can be created is the honest guarantee and blocks adoption on legacy stores. Promising only ordered grants ships everywhere and quietly leaves the strongest claim unbacked.

Decision

The service promises ordered, expiring grants for every class. It claims mutual exclusion only for classes declared fenced, and a class may be declared fenced only after the pause test passes against the real resource. A correctness class may be created as advisory, but only with a named risk owner and an expiry date, and it appears on a standing risk register until fenced or renewed on the record.

How it works on-premise

The LockClass CRD has required fields enforcement: fenced | advisory and, for advisory, riskOwner and waiverExpires. A validating admission webhook rejects a class that omits them. The class controller refuses to mark a class fenced unless a pause-test result object referencing that class and resource exists. ClickHouse produces the advisory register nightly from the class list and grant counts.

Options weighed

• ChosenOrdered grants everywhere, exclusion claimed only when fenced, advisory registered: Ships to every team on day one without letting any of them believe they are protected when they are not.
• RejectedRefuse correctness locks on any unfenced resource: Honest, and it sends the teams with legacy stores back to a Redis key with an expiry, which is strictly worse.
• RejectedPromise exclusion and document the pause caveat: A caveat nobody reads is the industry's current state. It is how duplicate postings happen in systems that have a lock.
• Right elsewhereRefuse locks entirely and push conditional writes: Right for an estate where every store supports compare-and-set; many do, and ADR-16 pushes teams there first.

Consequences

What it buys

• The platform's claim is exactly as strong as the evidence for it
• Unfenced risk is visible to security and has a date on it
• Adoption is not blocked on the slowest resource team

What it costs

• A second concept for every owner to understand
• Waiver administration and quarterly review
• Some advisory classes will be renewed indefinitely, and the register makes that visible rather than preventing it

Choose differently when

Flip to refusing unfenced correctness locks when every guarded store in the estate supports a conditional write and the register has been empty for two quarters. At that point advisory correctness is only a way to skip work.

Why it holds up over time

The distinction between an ordered grant and enforced exclusion is a law of distributed systems, not a feature of this stack. It will be true of whatever store replaces etcd, and the register keeps the gap visible through every reorganisation.

How is a 64-bit token minted that strictly increases per key, is never reused, and is committed with the grant?

Context

The obvious design is a counter key per lock, incremented in the grant transaction. It works, and the counter must then live forever, because deleting it lets the next grant restart at one. With four million distinct keys a day, the counter table becomes the largest thing in the store and a compaction problem nobody planned. The token must also stay monotonic across restores, where any stored value can go backwards.

Decision

The token is epoch × 2^48 + R, where R is the etcd revision at which the grant transaction committed and the epoch occupies the top bits, kept below 2^15 so the value is positive in a signed 64-bit column. Revisions are cluster-wide, strictly increasing and never reused within a cluster's life, which is stronger than per-key monotonicity. Gaps are expected and carry no meaning.

How it works on-premise

The arbiter reads header.revision from the successful TxnResponse and packs it with the epoch it compared in the same transaction. No additional key is written. The proto carries the token as a fixed64 plus the decoded (epoch, revision) pair for diagnostics only; resources compare the packed integer.

Options weighed

• ChosenEpoch plus commit revision: Minted by the commit itself, zero extra state, and a restore is handled once by the epoch rather than per key.
• RejectedPer-key counter key incremented in the grant transaction: Correct, but the counters can never be deleted and still need an epoch to survive a restore.
• Rejectedetcd lease ID as the token: Lease IDs are not ordered with respect to grants on a key, and many holds share one session lease.
• RejectedHybrid logical clock timestamp: Adds clock assumptions to the one property that must hold without them.

Consequences

What it buys

• No token state to compact or migrate
• Tokens from different keys are comparable, which simplifies debugging
• Header revision is already returned on every etcd response

What it costs

• Token values are large and non-contiguous, which surprises people expecting 1, 2, 3
• Token semantics are tied to one cluster's revision space; rehoming a key requires an epoch advance (ADR-09)

Choose differently when

Choose a per-key counter if the store behind the arbiter has no global monotonic commit index, for example a store that shards internally without exposing one.

Why it holds up over time

2^48 revisions last about 297 years at 30,000 commits a second, per epoch. The packed format is frozen in the proto contract, so resources never need to change the width of the column they store it in.

How is a restored lock log prevented from issuing a token it has issued before?

Context

Restoring a snapshot rolls revisions back to the snapshot point. Any grant made after the snapshot has handed a holder a token the restored cluster will issue again. The requirement says this must be impossible by construction, not by procedure. Restores happen during the worst hour of an operator's year, which is exactly when procedural steps get skipped.

Decision

A /sys/epoch record stores the current epoch and the etcd cluster ID it was minted for. Every grant compares the epoch key. The arbiter compares the cluster ID in every response header with the one in the epoch record and refuses to serve on mismatch. A restore always produces a new cluster ID, so a restored cluster is sealed until a single transaction sets epoch E+1 bound to the new ID and deletes every restored hold.

How it works on-premise

etcdutl snapshot restore generates a new cluster ID by design. The epoch-advance job is a Kubernetes Job gated by a two-person approval in the Admin API; it runs one Txn: compare epoch = E, put /sys/epoch = {E+1, newClusterID}, delete range /l and /q. New epochs are allocated from the coord cluster so no two clusters ever reuse one.

Options weighed

• ChosenCluster-ID-bound epoch with arbiter seal: The service cannot serve without the advance, so the advance cannot be forgotten.
• RejectedRunbook step to bump the epoch after restore: Correct on the day it is followed, and one skipped step is a duplicate token.
• RejectedEpoch derived from wall-clock time at startup: Puts a clock back into the one property designed to be independent of clocks.
• Right elsewhereNever restore; rebuild empty and let everyone reacquire: Acceptable if tokens are also epoch-advanced; the design effectively does this, keeping the snapshot for the shard map and class state.

Consequences

What it buys

• Token regression is impossible without deliberately editing the epoch key
• The seal is visible: arbiters report sealed status on every request
• Rehoming reuses the same mechanism

What it costs

• Every hold is released at once after a restore, producing a reacquisition storm
• A second person must be reachable during disaster recovery
• Relies on etcd continuing to assign a new cluster ID on restore, which the release gate checks

Choose differently when

Unnecessary on a store whose restore preserves a monotonic index beyond the snapshot, such as one with continuous log archiving to a point after the last committed entry. Such stores are rare and still benefit from the seal as a guard.

Why it holds up over time

The mechanism depends on one invariant, that a restored cluster is distinguishable from the original, and on nothing else. Any future store can supply an equivalent identity, and the arbiter check is ten lines.

When no majority is reachable, does the service keep granting?

Context

Correctness classes must never issue a second grant, so no grants and no renewals can happen without a quorum, and holders must self-fence as their deadlines pass. Efficiency classes, where two holders only waste money, would rather keep running. The requirement wants that difference declared and every open grant traceable afterwards.

Decision

Every class declares onQuorumLoss. Correctness classes may only declare closed. Efficiency classes may declare open, in which case an arbiter that cannot reach a majority grants from local memory with token zero, spools each grant to a local write-ahead file, and ships the spool to the audit store with unprotected = true on recovery. The MVP is closed for all classes.

How it works on-premise

The class admission webhook rejects enforcement: fenced or a correctness purpose combined with onQuorumLoss: open. The spool is a small append-only file on the arbiter pod's local persistent volume, shipped by the tailer's recovery step. The reconciliation list is a ClickHouse view filtered on unprotected over the outage window.

Options weighed

• ChosenClosed by default, open only for declared efficiency classes: Safety where it matters, availability where it is cheap, and a record of every compromise.
• DeferredClosed for everything: The MVP posture. Moves to the chosen option in Phase 3 once the reconciliation report exists.
• RejectedOpen for everything during outages: Converts a quorum outage into a correctness incident across every class at once.

Consequences

What it buys

• Quorum loss is an availability incident, not a correctness incident
• Efficiency workloads keep moving
• Every compromise is a bounded, queryable list

What it costs

• Correctness workloads stop during quorum loss
• Fail-open adds a local spool and a reconciliation path to build and test
• Owners must classify honestly; a misdeclared efficiency class is a silent correctness risk

Choose differently when

Flip a class to closed the first time anyone has to clean up after one of its unprotected grants. That cleanup is evidence the class was never an efficiency class.

Why it holds up over time

Safety-before-liveness is the one posture that never needs revisiting when the business grows; the only thing that changes is which classes deserve the exception, and that list lives in reviewed files.

Which replicated, linearizable store holds lease state and mints the ordering?

Context

The store needs linearizable conditional multi-key writes, leases whose expiry is a committed entry measured on the quorum's monotonic clock, a strictly increasing commit index, and a watch that resumes from a position. It has to be open source, run on the operator's hardware, and be operable by a platform team that already runs Kubernetes.

Decision

Use etcd on the 3.6 release line, run as dedicated clusters for lock traffic. The arbiter uses Txn for grants and releases, LeaseGrant and LeaseKeepAlive for sessions, header revisions for tokens and Watch for queues and audit. Lease checkpointing is enabled so a leader change carries remaining TTL forward.

How it works on-premise

Members run as a StatefulSet pinned to tainted nodes with local NVMe persistent volumes, guaranteed QoS and CPU pinning. --quota-backend-bytes is 8 GB, auto-compaction is periodic at one hour, defragmentation runs one member at a time off-peak. Client and peer TLS use certificates from a private CA; RBAC is enabled with per-prefix roles.

Options weighed

• Chosenetcd: Every required primitive is native, and its Raft implementation is exercised by every Kubernetes cluster in existence.
• RejectedApache ZooKeeper: Proven for exactly this job, with ephemeral sequential nodes and zxids. Loses on operational familiarity for a Kubernetes-native team and JVM tuning on the latency tail.
• RejectedConsul: Sessions and KV locks exist, but its lock-delay and session-invalidation semantics are coarser, and it brings a service-discovery product the estate does not need here.
• DeferredCustom Raft on hashicorp/raft or etcd-io/raft: Could remove the key-value layer's overhead at very high scale. Revisit only if a benchmarked shard cannot meet the per-cluster ceiling.
• Right elsewherePostgreSQL advisory locks or SELECT FOR UPDATE: Right when the guarded resource is itself one PostgreSQL database; the lock and the write then share a transaction and no service is needed.
• RejectedRedis with Redlock: Relies on bounded clock drift and process pauses for safety and has no committed expiry. The failure this service exists for is the one it does not handle.

Consequences

What it buys

• No new consensus implementation to trust
• Large pool of engineers who already operate etcd
• Transactions, leases and watches in one API

What it costs

• Every key carries etcd's MVCC overhead and compaction obligations
• Write throughput per cluster is bounded, so sharding comes earlier than with a purpose-built log
• etcd's keepalive path is leader-served rather than logged, which the design accounts for in ADR-10

Choose differently when

Choose ZooKeeper where a team already runs it well for Kafka or HBase and has no Kubernetes estate. Choose a purpose-built Raft log if sustained lock writes exceed what ten entity shards can carry.

Why it holds up over time

etcd is a CNCF-graduated project that Kubernetes cannot exist without, which makes it one of the safest open-source bets for continued maintenance. The arbiter uses four etcd capabilities behind an internal interface, so a replacement is an arbiter change, not a protocol change.

Where do the etcd members physically run?

Context

Grant latency is the fsync latency of the slowest member in the acknowledging majority. The Kubernetes control plane already runs an etcd, and reusing it is tempting. That etcd is sized and tuned for the API server, and a lock storm there degrades the cluster the lock service itself is deployed on.

Decision

Lock clusters are separate etcd clusters on dedicated worker nodes with local NVMe, one member per hall for the entity tier and a 2-2-1 split for coord. Disks are encrypted with LUKS and unlock through Clevis against Tang servers on a separate network.

How it works on-premise

Node labels lock-etcd=true and a NoSchedule taint; local persistent volumes through the local static provisioner; WAL and data on the same NVMe with nothing else on it; ionice and CPU manager static policy. Tang servers run in two halls; a node that cannot reach Tang at boot stays locked rather than falling back to a passphrase.

Options weighed

• ChosenDedicated clusters, dedicated nodes, local NVMe: Predictable tail latency and no shared fate with the orchestrator.
• RejectedReuse the Kubernetes control-plane etcd: Couples lock load to cluster health, and the API server's own leases would compete with lock traffic.
• Rejectedetcd on shared workers with network block storage: Network storage fsync tails of tens of milliseconds consume the whole latency budget.
• Right elsewhereetcd on bare metal outside Kubernetes: Equally valid where the platform team runs systemd services well; loses the uniform deployment and rollout tooling.

Consequences

What it buys

• Stable fsync latency and an isolated failure domain
• A stolen disk reveals nothing; a reboot needs no human
• Failure domains are explicit in the node placement

What it costs

• Dedicated hardware that sits well below capacity by design
• Tang becomes a boot-time dependency to operate
• Hall independence must be verified, not assumed

Choose differently when

Run on shared workers only in a non-production environment, or where measured fsync p99 on shared storage stays under 2 ms at burst.

Why it holds up over time

Separating a lock quorum from the orchestrator's own state is a boundary that stays correct however the orchestrator evolves, and local-disk consensus remains the lowest-latency option on any hardware generation.

One service for coarse and fine-grained locking, or two?

Context

Leader election needs tens of long-held, highly available locks. Entity locking needs hundreds of thousands of short, high-churn ones. On one cluster, entity churn inflates compaction, defragmentation and tail latency for the leader-election locks the rest of the platform depends on. Two clusters double the operational surface and require a routing rule that callers will eventually get wrong.

Decision

Same protocol, same client library, two tiers. The coord tier is a five-member cluster tuned for availability; the entity tier is sharded three-member clusters tuned for write throughput. The tier is a required field on the lock class, so routing is decided once by the owner in review and never by a caller.

How it works on-premise

The arbiter holds one clientv3 client per cluster and selects by class tier and, for entity, by the shard map (ADR-09). The coord cluster also stores epoch allocation and the shard map itself. In the MVP both tiers point at one cluster; the split is a configuration change once the coord cluster exists.

Options weighed

• ChosenTwo tiers behind one protocol, tier on the class: Blast radius isolation without asking callers to route.
• DeferredOne cluster for everything: Acceptable for the MVP's volume; split at Phase 2 or on the first latency incident traced to entity churn.
• RejectedTwo separate services with separate APIs: Two libraries and two sets of semantics for what is one primitive.

Consequences

What it buys

• Leader election survives an entity-tier incident
• Each tier tuned for its own shape
• No caller-side routing mistakes

What it costs

• More clusters to operate, patch and back up
• Moving a class between tiers is a rehome, not an edit

Choose differently when

Stay on one tier if the estate never exceeds a few thousand concurrent entity locks; the operational cost of a second cluster then buys nothing measurable.

Why it holds up over time

The tier is a property of a class, not of the infrastructure, so new tiers, such as a regional or a global coord, slot in without any existing class or caller changing.

What is the quorum topology across regions?

Context

A stretched quorum coordinates globally and pays inter-region latency on every acquisition, including the local majority. Regional clusters are fast and cannot coordinate a globally unique action. A hierarchy, regional clusters holding leases on a global lock, adds a second lease layer with its own expiry semantics and failure modes.

Decision

Each region runs its own coord and entity clusters, and no lock spans regions. A key's tenant namespace is homed to one region. A global lock is not offered until a named resource genuinely needs one, and then only as a separate stretched coord cluster with its acquisition latency, p99 around 150 ms, published before anyone commits to it.

How it works on-premise

Region is a field on the namespace. The arbiter in region A refuses keys homed to region B with a redirect error naming the region, rather than forwarding, so cross-region latency is never hidden inside a call.

Options weighed

• ChosenRegional only, global deferred and priced: Every local lock stays fast, and the expensive option is a decision rather than a default.
• RejectedStretched quorum across three regions: Every acquisition pays a cross-region round trip for the benefit of a small minority of locks.
• RejectedHierarchical regional and global leases: Two expiry layers whose interaction under partition is harder to reason about than either alone.

Consequences

What it buys

• Local latency for every lock
• A region's isolation cannot stall another region's locks

What it costs

• No globally unique action without a separate, slower cluster
• Namespaces must be homed deliberately

Choose differently when

Build the stretched coord cluster when a named resource, such as a global ledger migration or a cross-region singleton, cannot be partitioned by region and its owner accepts the published latency.

Why it holds up over time

Physics does not improve. Inter-region latency will be roughly what it is now in ten years, so a topology that keeps it out of the common path does not age.

How is the key space spread across entity shards, and how do keys move without token regression?

Context

A key must be deterministically homed to exactly one cluster. Adding a shard must move few keys. Tokens are revision-based and each cluster has its own revision space, so a key moved to a cluster with a lower revision would issue lower tokens than its old home did.

Decision

Entity keys are homed by rendezvous hashing over namespace and path prefix against a versioned shard map stored in the coord cluster. Adding a shard moves only keys whose highest score changes to it. Before the destination accepts moved keys, its epoch is advanced above the source's epoch, and the move proceeds only for ranges with no current holders.

How it works on-premise

The shard map is a versioned document in /sys on the coord cluster, watched by every arbiter. A rebalance controller drains a range by refusing new grants for it, waits for holds to release or expire, advances the destination epoch, flips the map version and resumes.

Options weighed

• ChosenRendezvous hashing with epoch-fenced moves: Minimal movement, no ring tokens to manage, and the same epoch mechanism as restore.
• RejectedConsistent-hash ring with virtual nodes: Works, with more state to balance and no advantage at tens of shards.
• Right elsewhereManual range assignment by namespace: Fine for a handful of very large tenants who want their own shard; supported as an override in the map.

Consequences

What it buys

• Adding a shard moves about 1/N of keys
• One mechanism for restore and rehome safety

What it costs

• A moving range is briefly unavailable for new grants
• The shard map is one more critical document on the coord cluster

Choose differently when

Prefer explicit per-tenant assignment when the estate is a small number of very large tenants whose isolation matters more than balance.

Why it holds up over time

Rendezvous hashing has no tuning parameters that drift with scale, and epoch-fenced moves make every future rebalance safe by the same rule that makes a restore safe.

How is renewal traffic kept from dominating the write path?

Context

250,000 held leases at a 15 s TTL renewed every 5 s is 50,000 renewals a second. Unbatched, that is four times the steady write target and more than any single etcd cluster should carry. The requirement asks for batched session renewal as a requirement, not an optimisation.

Decision

The client library opens one etcd lease per distinct (SPIFFE identity, TTL) pair and attaches every hold of that TTL to it. One LeaseKeepAlive renews all of them. A session's expiry revokes every attached hold in one committed entry. Renewal and expiry of individual holds do not exist as separate operations.

How it works on-premise

At 40,000 sessions with 5 s keepalives, renewal is 8,000 keepalives a second, handled by the etcd leader without a Raft proposal; lease checkpoints are proposed periodically so remaining TTL survives a leader change. The pinned etcd release must confirm leadership before honouring a keepalive, and the release gate injects leader isolation to verify that no deposed leader extends a lease.

Options weighed

• ChosenOne lease per session, holds attached: Batching falls out of the data model instead of being built as a feature.
• RejectedOne etcd lease per lock: Simple, and renewal cost scales with held locks rather than with processes.
• RejectedHeartbeat keys written by the application: Every heartbeat is a consensus write, and expiry becomes a timer decision on one node.

Consequences

What it buys

• Renewal cost scales with processes, not locks
• A crashed process releases everything in one entry
• Batched renewal of 500 holds costs the same as one

What it costs

• A hold cannot have its own TTL different from its session's; a class with a different TTL gets its own session
• Keepalives are not log entries, which departs from the requirement's wording and is covered by leadership confirmation and checkpoints instead

Choose differently when

Use per-lock leases when a caller holds one or two locks per process and needs different expiry per lock; the batching benefit is then negligible.

Why it holds up over time

Tying liveness to the process rather than to each resource it touches is how every long-lived coordination system has ended up, from Chubby sessions to ZooKeeper sessions. It will outlast the store.

How long is the lease, and who chooses it?

Context

A short TTL recovers faster from a dead holder but multiplies keepalive load and turns latency spikes into false expiries. A long TTL is cheap and leaves a crashed holder blocking progress for its whole duration. Whether a false expiry is harmless depends on whether the resource is fenced, so this decision depends on ADR-01.

Decision

The platform allows 5 s to 300 s with a 15 s default. Each class chooses within that range; callers cannot override per request. Fenced classes may go short, because a false expiry costs only a retry. Advisory classes have a floor of 30 s, because a false expiry there can produce a duplicate write. The class pull request prints keepalive load and liveness bound for the chosen value.

How it works on-premise

A CI check on the class repository computes expected sessions × keepalives per second from the namespace's current session count in Prometheus and comments on the pull request. The admission webhook enforces the range and the advisory floor.

Options weighed

• ChosenPer class, within a range, cost printed: The owner decides with the numbers in front of them, once.
• RejectedOne platform-wide TTL: Leader election and 2-second entity writes want different answers.
• RejectedCaller chooses per acquisition: Unreviewed, uncosted, and indistinguishable from a bug when a caller passes 300 s.

Consequences

What it buys

• Cost and liveness trade-off is explicit and reviewed
• False-expiry risk is tied to enforcement posture

What it costs

• Owners must understand the trade-off
• Different TTLs in one process mean more sessions

Choose differently when

Allow per-request TTL within the class range if a class guards work units with highly variable duration and fencing is in place, so a shorter request cannot cause damage.

Why it holds up over time

TTL is the primary cost lever of any lease system. Keeping it as a reviewed, costed configuration value means it can be retuned as hardware and networks change without any code change.

What does a holder do when it cannot renew but has not been told it lost the lock?

Context

Aborting at the first failure is safe and wastes work on every network blip. Continuing to a locally computed deadline trusts local timing, which a stop-the-world pause violates. Continuing until told is unsafe by construction, because a partitioned holder is never told.

Decision

Two stages in the client library. At the first failed keepalive, stop granting permission to start new units of work and let in-flight work finish. At the local deadline, call onFence. The deadline is the monotonic send time of the last successful keepalive plus the TTL minus a one-second margin. Using send time guarantees the local deadline precedes the leader's expiry for that keepalive.

How it works on-premise

The Go library uses time.Now() monotonic readings; Java uses System.nanoTime(); Python uses time.monotonic(). The application checks lock.canStart() before each unit of work, and onFence is a required constructor argument. The library also exposes remainingTTL() so a caller can decline to start a unit it cannot finish.

Options weighed

• ChosenTwo-stage drain then fence at a send-time deadline: Cheap under blips, conservative under partitions, and honest about pauses.
• RejectedAbort immediately on first failure: Safe, and turns every 200 ms blip into aborted work across the fleet.
• RejectedContinue until the service says the lock is lost: A partitioned holder is never told; the requirement names this as unsafe.

Consequences

What it buys

• Holder always believes it has lost before the log says so
• Transient failures cost a pause, not a redo

What it costs

• Applications must structure work into units that can check canStart()
• Does nothing for a pause that swallows the deadline, which only the token handles

Choose differently when

Abort immediately where units of work are cheap to redo and the resource is advisory; the conservative path then costs little.

Why it holds up over time

Monotonic clocks, send-time deadlines and a callback are available in every language runtime and will stay so. The rule is independent of the store.

Where is the re-entrant hold count kept?

Context

The requirement asks for re-entrant acquisition by the same session with a hold count, released at zero, and refusal when the same identity holds under a different session. Counting in the log costs a consensus write per nested acquire.

Decision

The library keeps the count per session and key and only calls the service for the outermost acquire and the final release. The arbiter refuses a grant when the key is held by the same SPIFFE identity under a different session lease, returning a leak error rather than a contention error.

How it works on-premise

The holder key's value carries the SPIFFE ID and session lease ID. The arbiter's grant transaction reads the existing value on failure and classifies the refusal. Re-entrancy is available from Phase 2.

Options weighed

• ChosenCount in the library, leak check in the arbiter: Zero consensus cost for nesting; the dangerous case is still caught server-side.
• RejectedCount in the holder key: Correct and costs two writes per nested call for no safety gain within one session.

Consequences

What it buys

• Nested acquisition is free
• Leaks across sessions are named as leaks

What it costs

• The count is lost if the library crashes, which is also when the session and all its holds end, so nothing is left inconsistent

Choose differently when

Move the count server-side if multiple processes ever share one session, which this design forbids.

Why it holds up over time

Keeping per-process bookkeeping in the process is the arrangement that stays cheap as call patterns change.

Is the wait queue durable state?

Context

Replicating the queue through consensus preserves arrival order across failures and puts every enqueue and cancellation on the write path, which for a hot key can exceed its grant traffic. In-memory queueing is nearly free and loses order when the queue's host fails.

Decision

Envoy ring-hashes on the lock key so every waiter for a key lands on the same arbiter, which queues them in arrival order in memory. Order is declared best-effort: it survives an etcd leader change and is lost on arbiter restart or ring rebalance, after which waiters retry. A class may set fairness: durable, which stores waiters as keys under /q ordered by create revision, at two consensus writes per waiter, printed on the class PR.

How it works on-premise

Envoy route uses hash_policy on the x-lock-key header with ring_hash load balancing across arbiter endpoints. Durable waiters use the etcd concurrency recipe: put /q/<key>/<leaseID> on the waiter's session lease and grant when its create revision is the lowest.

Options weighed

• ChosenIn-memory by default, durable per class: Most classes get fairness for free; the few that need it pay for it knowingly.
• RejectedDurable queue for every class: Doubles write load on hot keys to protect order that most callers do not rely on.
• RejectedNo queue; callers poll with backoff: Starvation under contention and a thundering herd on every release.

Consequences

What it buys

• No consensus cost for queueing on most keys
• One release produces one grant attempt

What it costs

• A waiter can lose its place on arbiter restart
• Key-affine routing means one arbiter's loss affects all waiters for its keys

Choose differently when

Make durable the default if contention reports show waiters routinely losing position during arbiter rollouts on correctness classes.

Why it holds up over time

The choice is a class field, so the default can move in either direction as hardware makes consensus writes cheaper, without any caller changing.

What may a release notification cause?

Context

Watch streams can drop, reorder around reconnects and lag. A design that treats a delete event as permission to hold turns a lost or late event into a double grant.

Decision

Every grant, including one triggered by a notification, is a full grant transaction comparing key absence and epoch. A waiter that misses a notification falls back to polling every second with jitter until its deadline. The watch stream is authoritative for nothing.

How it works on-premise

The arbiter watches the key from the revision returned by the failed grant, so no event between the check and the watch is missed. On watch cancellation due to compaction, it retries the grant transaction immediately and re-watches from the new revision.

Options weighed

• ChosenNotification as a hint, transaction as the grant: A lost event costs a second of delay, never correctness.
• RejectedGrant directly on delete event: Saves one round trip and makes safety depend on stream delivery.

Consequences

What it buys

• Watch failures are latency incidents only
• No special-case grant path to test

What it costs

• One extra transaction per contended release

Choose differently when

None within this design. A notification that grants is a different, weaker service.

Why it holds up over time

Separating hints from authority is independent of any watch implementation and survives every change to the notification transport.

Who holds the fencing high-water mark?

Context

Resource-side marks are cheapest and fastest but require every resource to implement the rule. A validation call back into the lock service centralises the rule and puts the service on every write path. Using the store's own conditional write is often strictly better, and raises the question of whether the lock was needed at all.

Decision

The pattern is the store's own atomic conditional write: UPDATE ... SET fence = $token WHERE id = $id AND fence < $token for relational stores, and conditional put with an ETag or version precondition for object stores. The ValidateToken API exists from Phase 2 only for resources that genuinely cannot store a mark, is rate-limited separately, and is reported per namespace as a dependency.

How it works on-premise

Reference implementations ship for PostgreSQL (a fence column updated in the same statement as the guarded write) and for S3-compatible stores including MinIO (version metadata checked with If-Match). ValidateToken is a linearizable read of the holder key comparing the presented token with the current grant.

Options weighed

• ChosenStore's conditional write as the fence: Atomic with the write, no extra hop, no new dependency on the lock service.
• DeferredValidation call on every write: Phase 2 fallback. Not atomic with the write unless the resource holds its own lock between check and write.
• RejectedSidecar proxy that fences writes: Protocol-specific per store and adds a component on every data path.

Consequences

What it buys

• Mutual exclusion enforced where the write lands, atomically
• The lock service stays off data paths

What it costs

• Each resource type needs its own implementation
• Owners discover some stores cannot do it, which feeds the advisory register

Choose differently when

Prefer the validation API for a resource type with low write volume, many instances and no conditional write, where one central rule is worth the hop.

Why it holds up over time

Conditional writes are a primitive of every serious store and are only becoming more common. A fence implemented as one outlives both the lock service and the store's current version.

How does a resource owner prove their resource is fenced?

Context

Owners will implement the fence subtly wrong: a check in one statement and the write in another, a less-than-or-equal instead of less-than, a mark per table instead of per record. A class marked fenced on assertion gives false confidence in exactly the case the platform claims to protect.

Decision

The platform ships the reference fence as copyable code and a test kit that acquires a lock, writes, freezes the holder with SIGSTOP past its TTL, lets a second holder acquire and write, resumes the first and asserts its write is rejected. A class can be marked fenced only when a result object for that kit run exists.

How it works on-premise

The kit is a container image run as a Kubernetes Job against the owner's staging resource. It writes a PauseTestResult custom resource that the class controller checks. Results expire after a year or on any change to the resource's schema version label.

Options weighed

• ChosenReference code plus mandatory pause test: The claim of fenced is backed by an observed rejection.
• RejectedDocumentation and code review only: Reviewers miss non-atomic checks.
• RejectedPlatform implements fences for every store: Unbounded scope and ownership of other teams' data paths.

Consequences

What it buys

• Fenced means tested
• The kit doubles as onboarding documentation

What it costs

• A staging resource is required
• Test results need renewal

Choose differently when

Relax to annual attestation for resources whose write path has not changed in a year and whose owners have passed the test before.

Why it holds up over time

A behavioural test survives rewrites of the resource, changes of language and changes of team. It checks the property, not the code that once had it.

Why run a service at all, rather than a client library that talks to etcd?

Context

A thick client is fewer moving parts and no extra hop. It also means 2,000 workloads hold etcd credentials, 40,000 sessions hold etcd connections, class rules and quotas are enforced in every language separately, and an etcd change requires every team to upgrade a library.

Decision

Callers speak lock.v1 gRPC through Envoy to a stateless Go arbiter, which is the only component with an etcd write credential for grants. The arbiter enforces epoch, class, authorisation, quota and queueing, and is stateless for correctness. Envoy terminates SPIFFE mTLS, applies per-identity local rate limits and routes by key.

How it works on-premise

Envoy runs as a Deployment of six replicas with SDS from the SPIRE agent. The arbiter is built with Go and etcd's clientv3, one client per cluster, nine replicas spread across halls. The proto is managed with buf, and breaking-change checks run on every pull request.

Options weighed

• ChosenArbiter service behind Envoy: One enforcement point, one etcd credential holder, and libraries that stay thin.
• RejectedThick client straight to etcd: Fine for one team; for 300 namespaces it scatters credentials and rules across every language.
• RejectedArbiter in Java or Rust: Both capable. Go matches etcd's own client and tooling, which matters most when debugging the client under faults.
• Right elsewhereHAProxy or NGINX as fan-in: Good gRPC proxies; Envoy wins on SPIFFE-native SDS and hash policy on a header.

Consequences

What it buys

• Rules enforced once
• etcd protected from connection count
• Libraries can be thin in every language

What it costs

• An extra hop of about 1 to 3 ms
• Two more tiers to deploy and monitor

Choose differently when

Use a thick client for a single-team platform with one language and a handful of services, where the arbiter's enforcement buys little.

Why it holds up over time

The gRPC contract is the stable boundary; everything behind it, including the store, can be replaced without callers upgrading. That is the property that keeps a shared service alive for a decade.

How is a workload identified, and how is it limited to its own keys?

Context

Every session and hold must be bound to an authenticated identity, and a workload may lock only keys it owns. Namespaces must not leak existence to one another through errors or timing. The authorisation question is narrow: may this identity lock under this prefix.

Decision

Workloads authenticate with X.509 SVIDs from SPIRE over mTLS. Authorisation is a list of prefix grants per SPIFFE ID, declared in the class registry and evaluated in-process by the arbiter as a longest-prefix match before any etcd read. Denials return one error, in constant time, whether or not the key exists.

How it works on-premise

SPIRE server runs HA with its datastore on PostgreSQL; agents run as a DaemonSet with Kubernetes workload attestation. SVIDs live one hour. Prefix grants are compiled into a radix tree on each class change.

Options weighed

• ChosenSPIRE plus in-process prefix grants: Attested identity and an authorisation check measured in microseconds.
• RejectedOpen Policy Agent sidecar: A general engine for a question that is a prefix match, on a path with a 40 ms p99.
• RejectedKubernetes service account tokens: Ties identity to one cluster and to bearer tokens rather than attested certificates.

Consequences

What it buys

• Identity is attested and short-lived
• Authorisation adds no network hop

What it costs

• SPIRE is a platform dependency to operate
• Complex conditions, such as time-of-day rules, are not expressible, deliberately

Choose differently when

Add a policy engine if authorisation needs attributes beyond identity and prefix, for example data classification of the guarded resource.

Why it holds up over time

SPIFFE is an open standard with multiple implementations, so identity survives a change of SPIRE, of orchestrator or of cloud. Prefix grants are data, portable to any future engine.

How can anyone other than the holder end a hold without making things worse?

Context

Leaked locks, holders alive and renewing but never releasing, are not automatically resolvable. Operators need a way to clear them. On an unfenced resource a force-release can let two writers in, and the operator must be told this before acting, not afterwards.

Decision

Force-release is available only through the Admin API, requires a requester and a different approver, both signed in through Keycloak with MFA and the lock-operator role, and a stated reason. The Admin API shows the class's fenced status before approval. The release is one transaction that deletes the hold and writes a /sys/force record, and may optionally revoke the holder's whole session.

How it works on-premise

Keycloak realm with a lock-operator role and required OTP or WebAuthn. The Admin API's etcd role permits delete on /l and put on /sys/force only. Force records carry a 24-hour lease; the tailer copies them to ClickHouse and the seven-year archive before they expire.

Options weighed

• ChosenTwo-person, reasoned, delete-only credential: Slow enough to prevent reflex, fast enough for an incident.
• RejectedSingle operator with audit: Audit explains the double write afterwards; it does not prevent it.
• Rejectedetcdctl access for SREs: Unbounded write access to the one store where a mistake is a duplicate grant.

Consequences

What it buys

• Every force-release carries who, why and whether it was safe
• No human holds a general etcd write credential

What it costs

• Needs a reachable second approver at night
• Keycloak outage blocks force-release, by design

Choose differently when

Allow a single-operator break-glass path with post-hoc review only if two-person approval has demonstrably extended incidents, and only for fenced classes.

Why it holds up over time

Two-person control with a recorded reason is how high-consequence operations have been governed for decades, and it survives any change in the tools that implement it.

How is grant, release, expiry and force-release history captured completely?

Context

An arbiter writing events alongside its grants has a dual-write problem: a crash between the two leaves a grant with no event. Worse, expiries are committed by etcd itself when a session lapses; no application code sees them, and they are the events investigators most need.

Decision

An audit tailer per cluster watches the whole lock key space from its last checkpointed revision and inserts events into ClickHouse keyed by (cluster ID, revision). Grant history is exported to Parquet on MinIO for a year; force-release records go to a bucket with object lock for seven years. The tailer alarms at half of etcd's compaction window and records any gap explicitly.

How it works on-premise

ClickHouse runs replicated with ReplacingMergeTree on (cluster_id, revision), making replays idempotent. The tailer is a singleton per cluster through an election on the coord cluster and checkpoints every few seconds. MinIO runs erasure-coded across halls with object lock in retention mode.

Options weighed

• ChosenRevision tailer into ClickHouse and MinIO: Complete by construction, including expiries, and idempotent.
• RejectedArbiter publishes events to Kafka: Misses expiries, dual-writes grants, and adds a broker the platform does not otherwise need here.
• Right elsewherePostgreSQL as audit store: Fine up to tens of millions of events; ClickHouse is chosen for scans over 90 days of several million events a day.

Consequences

What it buys

• Every committed change is captured, including expiries
• No broker on the grant path
• Cost and contention reports come from the same data

What it costs

• Tailer lag must stay well inside the compaction window
• ClickHouse is another stateful system to run

Choose differently when

Emit from the arbiter as well if audit must carry context the log does not have, such as denial reasons; keep the tailer as the source of truth for grants.

Why it holds up over time

A change-data-capture pattern keyed on the log's own position is independent of both the audit store and the arbiter, and survives replacing either.

What does the service emit, and with which labels?

Context

Operators need grant latency, contention ratio, hold times, expiry without release and renewal failures per namespace and class. Four million keys a day as a metric label would overwhelm any time-series store. Tokens are capabilities, and a token in a trace or log is a token anyone with log access can present.

Decision

Metrics use namespace, class, tier and cluster labels only. Per-key questions go to the inspection API. The client libraries never record the token in spans, logs or errors; the OTel Collector runs a redaction processor on lock.token as defence in depth. Expiry without release and longest hold are routed to owning teams as client-health signals.

How it works on-premise

Prometheus scrapes arbiters, Envoy and etcd; Grafana holds SLO dashboards; Alertmanager routes by namespace owner label. OTel Collector receives traces and logs and forwards to Tempo and Loki. A CI lint fails any metric definition that adds a key label.

Options weighed

• ChosenBounded labels plus inspection API: Time series stay cheap; per-key answers stay exact.
• RejectedPer-key metrics: Cardinality explosion in the first week.
• RejectedTokens in logs for debugging: Turns log access into write authority at guarded resources.

Consequences

What it buys

• Stable Prometheus cost as key volume grows
• No capability leaks through telemetry

What it costs

• Debugging one key requires the inspection API rather than a dashboard

Choose differently when

Add exemplars carrying key hashes if per-key latency investigation becomes routine; never raw keys or tokens.

Why it holds up over time

Bounded label discipline is what keeps any metrics backend affordable; OpenTelemetry keeps the emitting side portable across backends.

How is the zero-double-grant property protected against future changes?

Context

The property fails only under timing the unit tests never produce: partitions mid-transaction, a deposed leader accepting a keepalive, a pause at the wrong moment. The engineers who understand the design will not be the ones making changes in five years. etcd upgrades can change lease behaviour without any change in this codebase.

Decision

Every arbiter release and every etcd version change runs a suite on real clusters under Chaos Mesh faults: partitions, member kills, leader isolation, clock steps, disk latency and process pauses. Every client operation is recorded with start and end times, and Porcupine checks each history against a sequential lock model. One violation blocks the release and attaches the history.

How it works on-premise

The suite runs in a dedicated staging namespace on the same node types as production. Gitea Actions builds and signs images with cosign into Harbor; Argo CD promotes only digests that carry a passing gate attestation. etcd versions are pinned by digest.

Options weighed

• ChosenChaos Mesh plus Porcupine on every change: Checks the property itself on real software under the faults that break it.
• Right elsewherePeriodic external Jepsen analysis: Valuable as an annual deep review; too slow and costly to gate every release.
• RejectedUnit and integration tests only: Cannot produce the interleavings that matter.

Consequences

What it buys

• A regression in the core property cannot ship silently
• etcd upgrades are verified, not trusted

What it costs

• Release cycle lengthened by hours
• A realistic staging environment to maintain
• Checker model must be updated with the protocol

Choose differently when

None for correctness-class releases. Documentation-only or dashboard changes skip the gate.

Why it holds up over time

The gate encodes the design's central claim as an executable check, so the claim survives staff turnover, rewrites and store replacement.

Where do class definitions and prefix grants live, and who approves them?

Context

Classes carry the decisions that make locks safe or unsafe: enforcement, posture, TTL and tier. They must be reviewed by the resource owner, versioned, diffable and reversible, and they must not depend on an API only the platform team can call.

Decision

LockClass and LockGrant custom resources live in a Gitea repository with CODEOWNERS per namespace. Argo CD syncs them to each cluster; a validating webhook enforces required fields, TTL range and posture rules; arbiters watch them and keep the last good set if the Kubernetes API is unavailable.

How it works on-premise

One repository per organisation with a directory per namespace. CI prints renewal cost and liveness bound for each change. Arbiters cache classes in memory and serve from the cache during API outages.

Options weighed

• ChosenGit-reviewed CRDs via Argo CD: Review, history and rollback come from tools teams already use.
• RejectedAdmin API writes classes into etcd: Loses review workflow and puts configuration in the lock log's blast radius.
• RejectedConfigMaps: No schema validation or typed status.

Consequences

What it buys

• Every safety posture has an author, a reviewer and a date
• Configuration outlives the people who wrote it

What it costs

• Class changes take a pull request, not an API call
• Kubernetes API is a dependency for change, not for serving

Choose differently when

Offer an API for self-service efficiency classes if pull-request latency becomes a real adoption barrier; keep correctness classes in review.

Why it holds up over time

Declarative configuration in version control is the most portable form a decision can take; it survives a change of orchestrator, delivery tool or team.