Data Quality Service  ·  View 09 of 20  ·  Data

Storage Zones

Three zones, divided by what can be rebuilt and what would be gone for good.

Editable source SVG draw.io All views
Irreplaceable — losing it loses the ability to answer an auditor Verdict record Verdict store append-only · 25 mo Override audit log immutable · 7 yr Declared intent Rule catalogue Aurora · Git-sourced Tier + policy registry Aurora Rebuildable — derived, and cheaper to recompute than to restore Serving projections Current-state index DynamoDB · RTO 10 min Scorecard rollups nightly materialised Learned state Baseline store 28-day window · 13 mo Compiled plan cache disposable Custodial — other people's data, held under their classification Quarantine Quarantine tables Delta · 90 days Violation samples redacted by default Working state Debt register Aurora · mutable Verdict write buffer replayed on outage Delta Lake on S3 — the judged estate read-only to this service derives rows diverted Storage Zones — Divided by What Can Be Rebuilt Data store Security / platform Queue / topic event / async Quarantine carries the source dataset's classification and masking policy. It is a custodial copy of regulated data, not a lower-governance staging area. v 1.0 · owner Data Platform Architecture · date 2026-09

The division

  • Irreplaceable: the verdict record and the override audit log. Losing them loses the ability to answer an auditor about a period that has already passed (ADR-14).
  • Rebuildable: the state index, scorecards and baselines are projections. The state index has an RTO of 10 minutes because its recovery is a rebuild from verdicts, not a restore (ADR-04).
  • Custodial: quarantine holds other people's regulated data under their classification and masking policy. It is not a lower-governance staging area (ADR-15).

Numbers

  • Verdicts: 25 months retention, hot for the trailing 90 days. Sized to cover two audit cycles plus margin — stated assumption.
  • Quarantined rows: 90 days, then expired and counted as debt. Override audit log: 7 years, immutable.
  • Verdict store RPO 1 min / RTO 60 min; state index RPO 15 min / RTO 10 min.

Risks

  • Quarantine is the most likely place for a governance mistake: it is a copy of production data created by an automated process, in a table that no domain team thinks of as theirs.
  • Violation samples are the second: redaction is on by default, which means a single misclassified column is what stands between a verdict and a leak (ADR-14).
◀  Integration Surface All views Data Model  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records09-storage-zones