Distributed Lock Service  ·  View 19 of 26  ·  6 · Operations

Deployment — One Region, Three Failure Domains

Dedicated etcd members spread across three halls with independent power and network, beside the Kubernetes workers that run the stateless tiers.

Editable source SVG draw.io All views
Data centre A · on-premises Hall 1 · power + network domain coord members etcd × 2 entity-01 member etcd · NVMe · LUKS Worker nodes Envoy · arbiter Hall 2 · power + network domain coord members etcd × 2 entity-01 member etcd · NVMe · LUKS Worker nodes Envoy · arbiter Hall 3 · power + network domain coord member etcd × 1 entity-01 member etcd · NVMe · LUKS Worker nodes Envoy · arbiter Snapshot vault MinIO · data centre B Tang servers disk unlock keys etcdutl · 30 min Deployment — One Region, Three Failure Domains Data store Security / platform batch These are not the Kubernetes control plane's etcd. Losing any one hall loses no quorum. v 1.0 · owner Platform Architecture · date 2026-09

Decisions

  • Three halls, each its own power and top-of-rack domain. The entity cluster puts one member in each; the coord cluster splits 2, 2, 1. Losing any one hall leaves both with a majority.
  • etcd disks are LUKS-encrypted and unlock automatically through Clevis against Tang servers on a separate network. A stolen disk is unreadable; a rebooted node needs no human with a passphrase.
  • Snapshots are taken every 30 minutes and shipped to MinIO in a second data centre. They exist for the loss of a majority, not for routine recovery, which Raft already handles.

Assumptions

  • Inter-hall round trip under 1 ms. If the halls are really separate buildings with 3 to 5 ms between them, the latency budget on view 14 still holds, but it should be measured, not assumed.
  • Kubernetes 1.30 or later on the operator's own distribution. The design uses nothing beyond StatefulSets, taints, local persistent volumes and CRDs.

Risks

  • Shared hall infrastructure nobody has mapped, such as one core switch pair serving all three. Failure-domain claims are verified with a hall-down game day before go-live.