# Architecture One-Pager

*Distributed Lock Service · Solution Architecture v1.0 · Platform Architecture · 2026-09 · 26 views · open source, fully on-premises*

**A lock service cannot stop a process from writing. It can only make the second write rejectable. This architecture is built around that sentence.**

Schedulers run jobs that must not run twice, replica sets need exactly one leader, and workers mutate records no two workers may touch at once. Each team solves this alone today, usually with a database row, a Redis key with an expiry, or nothing, and each solution fails the same way: a holder pauses for longer than its lock lives, another process is granted the lock, and the first one wakes up and writes anyway. The failure is rare, silent and expensive, and it cannot be closed by a cleverer lock protocol, because a pause long enough to outlive the lease is indistinguishable from a crash.

The service issues leases, time-bounded grants that expire unless renewed, on dedicated etcd Raft quorums. Every exclusive grant returns a fencing token built from the cluster epoch and the revision of the transaction that granted it, so tokens are strictly increasing per key across expiry, leader change and restore. Holders present the token on every write, and a fenced resource refuses any token not higher than the last one it accepted. A stateless Go arbiter behind Envoy owns class rules, authorisation, quotas and wait queues; the client library owns renewal and tells the application to stop before its lease can have expired. Audit history is read off the log's own revision stream into ClickHouse. Every lock class declares whether its resource is fenced and what happens when the quorum is gone, and nothing about safety is left to a default.

## What it is, and what it is not

- **A service that issues ordered, expiring grants** — not a guarantee that two processes never act at once. That guarantee lives at the fenced resource.
- **A coordination primitive for correctness and efficiency locks** — not a transaction manager. No cross-key atomicity, no sagas.
- **Dedicated etcd clusters sized for lock traffic** — not the Kubernetes control plane's etcd, and not a general key-value store for teams.
- **A place to see who holds what and why it is stuck** — not a way to make force-release safe on a resource that does not check tokens.
- **Regional, with shards added behind one protocol** — not a globally consistent lock by default. A stretched quorum is priced and deferred.
- **Something the platform wants teams to use less** — not a free resource. Every grant is a consensus round trip, and the report says so.

## The decisions that are the architecture

1. **Promise ordered grants; claim exclusion only where the resource is fenced** (ADR-01) — The service never claims to prevent a second writer. Each class is declared fenced or advisory, and advisory correctness classes sit on a risk register with a named owner and an expiry date.
2. **The token is the epoch plus the commit revision** (ADR-02) — No per-key counter to keep forever. The revision of the granting transaction is strictly increasing and never reused within a cluster; the epoch covers everything a revision cannot.
3. **A restored cluster cannot serve until it advances the epoch** (ADR-03) — The epoch record is bound to the etcd cluster ID, and a restore always changes it. Token regression is prevented by construction, not by a runbook step someone has to remember.
4. **etcd on dedicated nodes, never the cluster's own** (ADR-05) — A mature, widely operated Raft implementation with leases, transactions and watches, on local NVMe in three failure domains, separate from the Kubernetes control plane.
5. **Renewal costs one keepalive per session, not per lock** (ADR-10) — Holds attach to a session's etcd lease, so batching is structural. 250,000 held leases cost 8,000 leader-served keepalives a second, not 50,000 consensus writes.
6. **The holder stops before the log says it has lost** (ADR-12) — Stop starting work at the first failed keepalive; fence at a deadline computed from the last keepalive's send time. The token handles the pause the library cannot see.
7. **The resource's own conditional write is the fence** (ADR-16) — A compare-and-set, a guarded UPDATE or an etag condition holds the high-water mark. A validation call back into the lock service is the fallback, not the pattern.
8. **Two tiers of cluster, chosen by the class** (ADR-07) — Leader-election locks on a small five-member coord cluster that entity churn cannot touch; entity locks on three-member shards. The caller never names a cluster.
9. **Audit is read from the log, never dual-written** (ADR-21) — The tailer follows etcd revisions into ClickHouse, so expiries, which no application sees, are recorded, and replays are idempotent on revision.
10. **No release without a checked history** (ADR-23) — Fault injection and a linearizability checker gate every arbiter release and every etcd upgrade. A single double grant in a recorded history blocks the release.

## Why this holds up over time

Lock services age badly in two ways: the store underneath stops being maintained, or the guarantees were tied so tightly to one implementation that replacing it means re-teaching every caller. This design is arranged so that the parts most likely to change are the cheapest to change, and the parts callers depend on are the least likely to.

- **The guarantee does not depend on the technology.** Fencing at the resource is a property of the write path, not of etcd. It was the right answer before Raft was published and will be the right answer after whatever replaces it. Every resource owner who implements the fence has made an investment no store migration can invalidate.
- **The token format has room for centuries.** 48 bits of revision last about 297 years at the burst rate per epoch, and 15 bits of epoch allow 32,767 restores or rehomes. The format is fixed in the proto contract, so no migration will ever ask resources to widen a column.
- **The store sits behind a four-operation contract.** The arbiter needs a conditional multi-key transaction, a lease that expires by committed entry, a revision that only goes up, and a watch from a revision. etcd provides them today; FoundationDB, a purpose-built Raft log or a future store could provide them tomorrow, and callers would not notice.
- **Growth is additive, never a re-platform.** Coord split, entity shards and regional clusters are all more clusters behind the same protocol. Each step is triggered by a measured threshold and fenced by an epoch advance. No growth step changes what a caller sends or what a token means.
- **Correctness is re-proven, not remembered.** The fault-injection and history-checking gate runs on every change, including every etcd upgrade. Engineers who were not present for the design still cannot ship a regression in the one property that matters.
- **Every component is replaceable and none is exotic.** etcd, Envoy, SPIRE, Keycloak, ClickHouse, MinIO and Prometheus are each widely operated, CNCF-graduated or foundation-governed, and replaceable by a peer without touching the lock contract. The in-house code is small: an arbiter, a tailer and client libraries.
- **The declared posture survives staff turnover.** Fenced or advisory, closed or open, TTL and tier are written in reviewed class files with the cost printed beside them. The reasoning survives the people who made it, which is the usual way safety properties decay.

## Non-functional targets

Targets from the requirement, with the mechanism that meets each one. Latency figures assume a three-member quorum across three halls with sub-millisecond round trips and enterprise NVMe, and are to be confirmed by benchmark at burst load before build.

| Quality | Target | How it is met | View |
|---|---|---|---|
| Simultaneous exclusive grants | Zero, not a budget | Grant is one Raft transaction comparing key absence and epoch; release deletes only on the holder's lease; release gate checks recorded histories under faults | 14 |
| Token regression | Zero, including restore and rebuild | Token = epoch · 2^48 + revision; epoch bound to cluster ID; restored cluster sealed until a committed epoch advance | 13 |
| Uncontended acquire | p50 ≤ 8 ms, p99 ≤ 40 ms | Warm mTLS connections through Envoy; one Raft commit with majority fsync on local NVMe; no extra key write for the token | 14 |
| Session renewal | p99 ≤ 25 ms; 500 holds in one call | One etcd LeaseKeepAlive per session renews every hold attached to it; served by the leader without a log entry | 11 |
| Watch notification | p99 ≤ 50 ms after release commits | Arbiter watches from the observed revision and grants the queue head; lost events fall back to jittered 1 s polling | 16 |
| Leader failover | RTO ≤ 3 s, no lease expires during it | etcd election timeout tuned for sub-millisecond halls; a new leader extends lease deadlines; lease checkpointing keeps remaining TTL honest | 26 |
| Full cluster rebuild | RTO ≤ 15 min | 30-minute snapshots in a second data centre; scripted etcdutl restore; epoch advance releases every hold and callers reacquire with jitter | 13 |
| Crashed-holder liveness | Re-grantable within TTL + 3 s | Session TTL 15 s at three missed 5 s keepalives; revocation is a committed entry; figure printed on every class PR | 18 |
| Acquisition availability | ≥ 99.95% monthly per cluster | Majority survives one hall; arbiter and Envoy stateless across halls; shed acquisitions before renewals under storm | 19 |
| Throughput | 12,000 writes/s steady, 30,000 burst | Renewals removed from the write path; queue head granted once per release; shards added past 60% of benchmarked ceiling | 22 |
| Audit durability | RPO ≤ 60 s, RTO ≤ 4 h | Tailer checkpoints revision every few seconds; replays idempotent on (cluster ID, revision); archive in MinIO | 12 |

## Scope

**In scope**

- Hierarchical keys by tenant and namespace; exclusive leases with fencing tokens in the MVP, shared mode in Phase 2
- Session model on etcd leases with keepalive, and expiry of every hold on session loss
- Try, bounded wait and watch acquisition modes; in-memory FIFO per key, durable queue as a class option
- Client libraries with a mandatory fence callback, local deadline and transparent reconnect
- Lock classes as reviewed configuration: TTL range, mode, enforcement, quorum-loss posture and tier
- Inspection API, two-person force-release, per-namespace quotas, audit trail and the advisory-lock risk register
- A reference fence for PostgreSQL and for S3-compatible object stores, with a pause test kit

**Explicitly out of scope**

- Enforcing exclusion inside guarded resources; the platform supplies the rule and the reference, owners implement it
- Distributed transactions, cross-key atomicity, sagas
- Leader election for the Kubernetes control plane itself
- Cross-region locks in the MVP; a stretched quorum only when a genuinely global resource is named and priced
- Use as a general configuration or service-discovery store

## The three-week proof

Not a slice of every feature. The proof shows the one property everything rests on: that a paused holder's write is rejected, and that no recorded history under injected faults contains two overlapping exclusive grants. If that cannot be shown on the operator's own hardware in three weeks, nothing else in this document matters.

1. One three-member etcd cluster on the target NVMe nodes across three halls, benchmarked at 30,000 transactions a second of mixed grant and release
2. The arbiter with Acquire, Release, session keepalive and the epoch check, behind one Envoy
3. The Go client library with the local deadline and onFence
4. The PostgreSQL reference fence on a real table
5. A Chaos Mesh schedule and a Porcupine model of the exclusive lock

- SIGSTOP a holder for 30 s with a 15 s TTL: the resumed write is rejected and onFence fires
- Partition the leader mid-grant: no history contains two holders, and grants resume within 3 s
- Destroy the cluster, restore a 30-minute-old snapshot: no grant is served until the epoch advances, and the first new token exceeds every token issued before the disaster
- Kill 2,000 clients at once and restart them: the reacquisition storm settles without a renewal being shed

## Open risks, carried rather than hidden

| Risk | If it lands | Response |
|---|---|---|
| Guarded resources that cannot be fenced | Correctness locks in front of them promise ordered grants and nothing more. A long pause can produce a duplicate write, and the business may believe it is protected | Advisory waivers with a named owner and expiry date, reported quarterly to security; the reference fence and pause kit lower the cost of fixing the resource (ADR-17) |
| Teams lock per record instead of per batch | Write load grows with business volume rather than with coordination need; the platform scales a design mistake | Per-record lock alarm and the monthly contention report; the platform publishes the cost and declines to shard for a namespace until the owner has reviewed it (ADR-11) |
| etcd behaviour changes across a release | Lease or keepalive semantics shift subtly and a safety property silently weakens | Every etcd upgrade passes the same fault-injection and history-checking gate as an arbiter change; versions are pinned, never floated (ADR-23) |
| Failure domains that are not independent | One switch pair or power feed takes down two halls and both quorums with them | Map shared infrastructure before build; hall-down game day before go-live; coord cluster can be moved to a second building if the map shows it is needed (ADR-06) |
| Operator skill with Raft under pressure | A well-meaning manual intervention during quorum loss, such as force-new-cluster on the wrong member, creates the duplicate grant the design otherwise prevents | Scripted recovery only, epoch seal enforced by the arbiter, and a quarterly restore drill on a staging cluster (ADR-03) |

The reasoning behind every component and technology choice is in the [Architecture Decision Record](decision-record): 24 records across 7 areas, each with the alternatives that lost and what the choice costs.