Data Quality Service  ·  View 06 of 20  ·  Structure

Layered Architecture

Seven layers, and what each of the middle ones is forbidden to know.

Editable source SVG draw.io All views
Experience Quality console state · scorecards Debt register UI owners · expiries State API read-only Rule CLI dry-run · lint Authoring Rule DSL declarative Profiling proposer suggests bounds Contract importer from catalogue Review + promotion owner approval Control Rule catalogue versioned Tier + policy registry who may block Scheduler priority lanes Plan compiler bounded cost Reconciliation sweeper every 10 min Evaluation Metadata evaluator no scan Scan evaluator pushed down Distribution evaluator vs baseline Reconciliation runner control totals Backfill lane low priority Verdict + state Verdict store append-only Current-state index hot read path Baseline store 28-day window Blast-radius resolver lineage walk Enforcement Circulation gate promote / hold Quarantine diverter row-level only Override service attributed · expiring Platform Unity Catalog grants + lineage Delta Lake on S3 the judged estate Observability MTTD · FP rate Audit log immutable · 7 yr publish plan verdict derive state promote Layered Architecture — and What Each Layer May Not Know Application we own Interface / broker Security / platform Data store Decision point synchronous event / async The evaluation layer may not know who owns a dataset or what a verdict will cause; the enforcement layer may not recompute one. That separation is what lets history be re-judged without re-running enforcement. v 1.0 · owner Data Platform Architecture · date 2026-09

The forbidden knowledge

  • The evaluation layer does not know who owns a dataset or what a verdict will cause. That is what lets the same evaluator be re-run over history without re-running enforcement (ADR-01).
  • The enforcement layer does not recompute a verdict; it reads one. A gate that could re-evaluate would be a second, divergent judge.
  • The authoring layer cannot write to the catalogue directly — publication goes through review and the tier policy, which is where 'who may block' is decided (ADR-13).

Assumptions

  • The rule DSL's source of truth is Git; the catalogue is its materialisation, rebuildable from the repository.
  • 40,000 active assertions at steady state across 12,000 datasets.

Deliberate omissions

  • Six of roughly forty layer-crossing edges are drawn — the spine from publication to promotion. The rest would obscure it.
  • The experience layer is sketched, not designed: this is a platform architecture, and the console's own information design is a separate piece of work.
◀  Journey — Turning a Rule On All views Container View  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records06-layered-architecture