Distributed Lock Service · View 13 of 26 · 4 · Data
Decisions
- The epoch record stores the etcd cluster ID it was minted for. etcdutl snapshot restore always creates a new cluster ID, so a restored cluster is detectably not the one that issued the old tokens.
- Every grant transaction compares the epoch key. An arbiter that sees a cluster ID mismatch in the response header refuses to serve, and the only accepted write is the epoch advance.
- The advance is one transaction that sets epoch E+1 bound to the new cluster ID and deletes every restored hold. Holders from before the disaster re-acquire and get tokens above anything they could have seen.
Why not a runbook step
- The restore happens during the worst hour of someone's year. A step that must be remembered under that pressure will be skipped once, and once is a duplicate token. A service that cannot start serving without the step cannot skip it.
What it costs
- After a restore every lock is released at once and every holder reacquires. Expect a reacquisition storm; jittered reconnect in the client is what keeps it survivable.
