Storage Tiering Service  ·  View 06 of 31  ·  2 · People and journeys

Journey: Reverse a Bad Policy at 03:10

Policy v43 is sending a population to tape that is still being read. On-call has to stop it, find exactly what it moved and put it back without causing a second incident.

Editable source SVG draw.io All views
Storage SRE on-call Goal — Stop policy v43 and put back only what it moved Trigger — Wrong-tier read rate alert at 03:10 Done when — Population restored; incident cost stated in dollars 1 · Alert 2 · Contain ◆ moment of truth 3 · Identify 4 · Reverse ◆ moment of truth 5 · Account 6 · Learn What they do Paged on wrong-tier rate Freezes policy v43 Lists moved population Starts reversal job Reads incident cost Adds a shadow check System behind it Alertmanager Ring controller Decision records Planner · reversal Cost ledger Policy repo How it feels In control Uneasy Under pressure Where it hurts 3.1 M objects moved Are rings still rolling? Some sources released Early-delete charges What the platform gives Ring stopped at 10% One flag halts all rings Population by version Under 24 h: pointer flip Charges per decision Regression becomes a gate Journey — Storage SRE Reverses a Bad Policy at 03:10 v 1.0 · owner Storage Platform Architecture · date 2026-09

The worst moment

  • Contain. The on-call engineer does not yet know how far the policy got. Rings, per-window caps and a single freeze flag mean the answer is bounded before anyone looks: at most the current ring, at most 2% of objects.
  • Reverse. Objects moved more than 24 hours ago have already released their source. Those need a real promotion with a real retrieval cost, and that cost is the honest measure of the incident.

What the architecture owes this journey

  • Every movement carries the decision id, and every decision carries the policy version, so 'what did v43 move' is one query (ADR-19).
  • Reversal is itself a planned policy with guards, rate limits and a budget. An emergency is the worst time to bypass the rules that stop a tool relocating the corpus.
  • The incident is closed with a dollar figure: retrieval, early-delete and movement charges attributed to v43.

Numbers

  • Freeze to halted rings under 30 seconds. Pointer-flip reversal of the last 24 hours of movements at 5,000 commits per second per shard group.
◀  Journey: Recall Seven Years of One Custodian All views Layered Architecture  ▶
Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises06-journey-reverse-a-classifier-regression