Distributed Lock Service  ·  View 05 of 26  ·  2 · People and journeys

Journey — On-call SRE Clears a Stuck Lock

A holder that is alive, renewing and never releasing, found at night and cleared without a double write.

Editable source SVG draw.io All views
On-call SRE night rota Goal — Unblock the queue behind one key without a double write Trigger — Longest-hold alarm on the orders namespace Done when — Queue drains and the old holder's writes are refused 1 · Paged 2 · Inspect 3 · Judge safety 4 · Force-release ◆ moment of truth 5 · Verify 6 · Review What they do Acknowledges page Looks up the key Reads fenced flag Requests release Peer approves Watches queue drain Files holder bug System behind it Alertmanager Inspection API Class registry Admin API + Keycloak Lock log watch Audit store How it feels In control Uneasy Stressed Where it hurts Holder alive, not working Advisory: release unsafe Approver hard to reach What the platform gives Runbook in the alert Holder, token, age, queue Safety stated, not implied Token advances on release Stale write rejected Trail with reason Journey — On-call SRE Clears a Stuck Lock v 1.0 · owner Platform Architecture · date 2026-09

The trough, and the answer to it

  • The worst moment is judging safety. A force-release on an advisory class can let two writers in, and a tired engineer under pressure will release anyway unless the tool says so plainly.
  • The console shows the class's fenced flag before the second approval is requested and records it with the release. The operator is told the truth at the point of decision rather than in the post-incident review.

Decisions it forces

  • The inspection API returns holder identity, session, grant revision, token, remaining TTL and queue depth for one key. The page links straight to it, pre-filled.
  • Force-release is a committed deletion whose revision becomes the floor for the next token. It also revokes the holder's session if the operator chooses, so a leaking process's library fences on its next keepalive.
  • The alert is the longest hold per namespace, not the error rate. A leaked lock produces no errors; it produces a queue.

Assumptions

  • A second approver on the on-call rota is reachable within ten minutes. Where that fails, the escalation is a named incident commander, never a single-person override.
◀  Journey — Service Owner Onboards a Correctness Lock All views Layered Architecture  ▶
Solution Architecture v1.0 · Platform Architecture · 2026-09 · 26 views · open source, fully on-premises05-journey-clear-stuck-lock