Distributed Lock Service · View 07 of 26 · 3 · Structure
Decisions
- etcd runs on dedicated nodes with local NVMe, not on shared Kubernetes workers with network storage. The fsync latency of the slowest acknowledging member is every grant's latency.
- Nine arbiter replicas behind six Envoy replicas. Envoy ring-hashes on the lock key, so all waiters for one key queue on the same arbiter and in-memory FIFO order means something.
- One audit tailer per etcd cluster, singleton by its own coord-cluster lease. Two tailers would double-insert; ClickHouse deduplicates on revision, so a brief overlap during failover is harmless.
Sizing assumptions
- Entity cluster: 12,000 writes per second steady, 30,000 burst, on three members with enterprise NVMe. To be confirmed by benchmark before build, not taken from a vendor chart.
- Arbiter: roughly 4,000 requests per second per replica at under 2 ms of added latency, dominated by TLS and the etcd client, not by lock logic.
- etcd backend quota raised to 8 GB against a live state held under 2 GB.
Risks
- A noisy neighbour on an etcd node. Nodes are tainted for etcd only, CPU is pinned, and the WAL disk carries nothing else.
