Storage Tiering Service  ·  View 30 of 31  ·  7 · Assurance

Reconciliation and Drift

Five ways the catalogue and storage can disagree, which one is believed about what, and what each disagreement becomes.

Editable source SVG draw.io All views
Found by Believed for where bytes are Action Recorded as Copy exists, commit lost Stuck in-flight row Tier Adopt if checksum matches Expected · paid twice Bytes with no row Rolling tier scan Tier Quarantine 35 d then gate deletes Orphan Class changed out of band Bucket notification Tier Correct row · revert Governance finding Row names missing bytes Read 404 · scan Tier Search replicas · mark unreadable Durability incident Checksum mismatch Sampled scrub Neither alone Repair from DC-B copy Durability incident Reconciliation — Every Kind of Disagreement, and Who Is Believed The tier is believed for where the bytes are; the catalogue for where they should be. Drift nobody can explain is a defect, never a reconciliation outcome. v 1.0 · owner SRE · date 2026-09

Decisions

  • The tier is believed for where the bytes are. The catalogue is believed for where they should be. Neither is trusted alone, and a disagreement is a finding with a cause, not a preference to settle quietly (ADR-31).
  • RGW lifecycle transitions are disabled on every placement bucket. A class change the platform did not decide is detected from bucket notifications and reverted per policy, because a storage layer moving objects on its own is a governance failure.
  • Orphaned bytes are quarantined for 35 days before the release gate deletes them. Paying twice for a month is cheaper than deleting something a lagging commit was about to name.

Numbers

  • A full rolling scan of every tier every 14 days. Unpacked objects are listed; packs are read by manifest, one small object per pack, not one request per member.
  • The same scan machinery rebuilds the whole catalogue in about 30 hours, most of it writing 9.2 billion rows.

Risk

  • A scan every 14 days means an out-of-band change missed by notifications can go unnoticed for up to two weeks. Reads are unaffected, because the catalogue still names bytes that exist; the saving figure is what is briefly wrong.
◀  Short-Lived Credentials All views Failure Modes  ▶
Solution Architecture v1.0 · Storage Platform Architecture · 2026-09 · 31 views · open source, fully on-premises30-reconciliation-and-drift