Distributed Lock Service  ·  View 23 of 26  ·  6 · Operations

Lock Hygiene Loop — Designing Locks Out

The monthly loop that turns contention, long holds and per-record locking into design changes rather than capacity.

Editable source SVG draw.io All views
Signal longest hold · rate Inspect holder · token · queue Classify leak · pause · design Act fix client or release Verify queue drained Design out CAS instead of lock Lock review monthly alert evidence cause change cheaper path? new baseline Lock Hygiene Loop — Designing Locks Out Security / platform Application we own Decision point Opportunity v 1.0 · owner SRE · date 2026-09

Decisions

  • Each namespace owner receives a monthly report: acquisitions, contention ratio, hold-time distribution, expiry without release, renewal cost and the class list with fenced status.
  • A class whose contention ratio stays near zero is flagged for removal. A lock nobody contends for is a consensus round trip bought for nothing.
  • An acquisition rate that tracks the owning service's record-write rate is flagged as a lock per record. The fix is batching or a conditional write, and the platform says so rather than adding a shard.

Why it is a loop

  • The platform's cheapest capacity is the locks it persuades teams to delete. Without the loop, every inefficiency becomes a scaling requirement for the platform team.

Assumptions

  • Contention and cost figures are computed in ClickHouse from the audit trail. No additional instrumentation is needed on the grant path.
◀  Growth Path — Scaling Out Without Re-Platforming All views Security — Trust Zones  ▶
Solution Architecture v1.0 · Platform Architecture · 2026-09 · 26 views · open source, fully on-premises23-lock-hygiene-loop