Distributed Lock Service  ·  View 26 of 26  ·  7 · Assurance

Failure Modes

Seven failure classes, how each is detected and contained, and which mechanism keeps the zero-double-grant property while it happens.

Editable source SVG draw.io All views
Detect Contain Safety held by Recover Leader failure Heartbeat timeout Grants pause ≤ 3 s Raft term · leases extended Re-election Quorum loss No majority ack Fail closed Holders self-fence Restore + epoch advance Holder pause > TTL Session lapses Re-grant, higher token Resource high-water Stale write rejected Snapshot restore Cluster ID mismatch Sealed · no grants Epoch in every Txn Advance to E+1 fsync failure WAL error Member stops, exits Majority still durable Replace member Hot-key herd Queue depth Reject past depth One Txn per release Owner redesign Restart lease storm Session open rate Jittered reconnect Shed acquire before renew Rate returns to base Failure Modes — Detection, Containment, What Keeps It Safe Every row fails towards refusing a grant. None fails towards issuing a second one. v 1.0 · owner SRE · date 2026-09

Decisions

  • Every containment fails towards refusing grants. The service is designed to be unavailable before it is wrong.
  • etcd's own mechanisms are used where they already give the required property: a leader change extends lease deadlines rather than expiring them, and a member whose WAL write fails exits instead of acknowledging.
  • A restart storm sheds acquisitions before renewals. Letting existing holders keep their sessions is cheaper than re-granting them.

Not in the grid, still covered

  • Clock step on a member: lease expiry is measured on the leader's monotonic clock, so a wall-clock step moves no deadline. Members are still ejected past 500 ms of skew because election timing assumes it.
  • Lost watch event: the waiter polls every second with jitter. Delay, never a false grant.
  • Compaction or defragmentation stall: defragmented one member at a time off-peak, under the same alarm as a leader failure.

Liveness published per class

  • A lock held by a crashed process is re-grantable within session TTL plus election window: 15 s + 3 s at the default. The class PR prints this figure.