File Upload & Scanning Pipeline  ·  View 11 of 21  ·  Data

Storage Classes

Three classes ordered by what loss costs, and only one with an RPO worth arguing about.

Editable source SVG draw.io All views
Class 1 — Precious: cannot be recomputed Object bytes Available objects ZRS in region, GRS cross Quarantine evidence immutable, 180 d Record of why Transition log append-only, 7 y, RPO 0 Verdict store hash × engine ver, RPO 0 Class 2 — Rebuildable from Class 1 Projections Object metadata rebuild from log Dedup index rebuild from verdicts Quarantine queue view derived Class 3 — Expendable: loss costs a retry In-flight and ephemeral Session chunk state RPO 60 s, one restart Scan work queue at-least-once, replayable Edge cache revalidated, available only Storage Classes — Ordered by What Loss Costs Only Class 1 has an RPO worth arguing about. The design earns that by refusing to let any Class 2 store be authoritative for anything — which is also why a corrupt metadata partition is a rebuild rather than an incident. v 1.0 · rpo 0 for class 1, 60 s for class 3

Decisions

  • Class 1 is the object bytes and the record of why they are where they are: both at RPO 0, synchronously replicated in region and asynchronously across region at ≤ 60 s
  • No Class 2 store is authoritative for anything, which makes a corrupt metadata partition a rebuild rather than an incident
  • Session chunk state is deliberately Class 3: losing it costs one restarted upload and no data, which is why it is allowed an RPO of 60 seconds

What this buys

  • Object metadata, the dedup index and the quarantine queue view are all rebuildable from the transition log and the verdict store
  • RTO 15 minutes for the read path in the secondary region, 60 minutes for full scan capacity, with verdicts replicated so a failover does not re-scan 12 PB
  • Object durability ≥ 11 nines, and zero tolerated cases of an object reaching a reader without a current clean verdict

Stated assumptions

  • 12 PB under management, growing 4 PB/year net of deletion
  • Transition log and verdict provenance retained 7 years, immutable
  • Quarantine evidence 180 days, then irreversibly destroyed
◀  Data Flow — One File to the First Read All views Data Model  ▶
Solution Architecture v1.0 · Microsoft Azure · Integration Platform Architecture · 2026-1011-state-classes