Storage Tiering Service  ·  View 31 of 31  ·  7 · Assurance

Failure Modes

Nine named failures, how each is detected, what contains it, what a read sees, and how the system returns to normal.

Editable source SVG draw.io All views
Detected by Contained by Reads see Back to normal Mover lost mid-copy Lease expiry No catalogue change Nothing Restart · partial deleted Telemetry stalls Canary age · Healthchecks Demotion stops Nothing Two fresh cycles Classifier regression Ring gate wrong-tier rate Ring · 2% window cap Some slower reads Pointer flip within 24 h Retrieval storm Queue depth Budgets · drive reservations Longer ETA Queue drains Shard primary lost VTOrc Semi-sync replica promoted ≤ 60 s errors, one shard Automatic DC-A lost Site monitoring DC-B runbook ≤ 15 min outage Sources still held 24 h Catalogue corrupted Reconciler · checksums Tier scan rebuild Errors in affected range ≤ 30 h · history lost Budget ends mid-recall Budget authority Job paused · staged kept Longer ETA Approver resumes Pack members lost Ceph unfound objects Members marked unreadable Explicit error never just slow Storage incident Failure Modes — Detection, Containment, What Reads See With classifier, planner and movers all down, every object reads at its current placement and every recall in flight completes. v 1.0 · owner SRE · date 2026-09

The posture

  • Fail static. With the classifier, planner and movers all down, every object reads at its current placement and every recall already in flight completes.
  • A mis-tiered object is a cost event; an unresolvable placement is an availability event. The first is budgeted in dollars and the second in minutes, and the design never trades one for the other.

The failures that are not errors

  • An early-delete charge. A commit lost after a successful copy, which costs a duplicate for a day. A retrieval storm. Each is expected, attributed and reported, and none pages anyone.

The failures that are incidents

  • A catalogue shard or site outage, bytes missing at a committed location, and lost pack members. The last is surfaced as an explicit unreadable state, never as a slow read that eventually times out.
◀  Reconciliation and Drift All views
Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises31-failure-modes