Data Quality Service  ·  View 02 of 20  ·  Context and scope

High-Level Architecture

Seven stages from a declared rule to an answer a consumer can act on.

Editable source SVG draw.io All views
1 · Declare Rule DSL in Git versioned assertions Tier + policy registry who may block 2 · Trigger Table-version commit event stream Schedule 5 min – hourly Reconciliation sweep catches missed events 3 · Plan Plan compiler fuse per scan Cost estimator budget per dataset 4 · Evaluate Metadata evaluator no scan Scan evaluator pushed down Distribution evaluator vs baseline 5 · Record Verdict store append-only · 25 mo Current-state index 4k reads/s 6 · Gate Circulation gate promote or not Quarantine divert ≤ 2% of batch 7 · Answer Consumers BI · features · reports Signalling pages · debt · scorecards derives known-good version Data Quality Service — High-Level Architecture Application we own Security / platform Queue / topic Data store Decision point External / third party event / async synchronous Stage 5 is the pivot: the verdict is written against a data version, not a run — which is what makes stage 4 re-runnable over history. v 1.0 · owner Data Platform Architecture · date 2026-09

Why this shape

  • Stage 5 is the pivot of the whole design: a verdict is recorded against a data version, which is what makes stage 4 re-runnable over history rather than only over today (ADR-01).
  • Three triggers, not one. The commit event stream is an optimisation; the schedule and the reconciliation sweep are what make a missed event a delay rather than a silent pass (ADR-09).
  • The plan stage exists so that cost is bounded before execution rather than discovered during it — an assertion that will not compile to a bounded plan is refused at authoring time (ADR-06).

Numbers

  • 180,000 assertion evaluations/day steady state; 9,000/hour peak in the nightly window; 4× burst for 90 minutes on a large backfill.
  • Verdict published within 90 s p95 of a Tier 1 commit; state read ≤ 20 ms p99.
  • All stated assumptions from the requirement, chosen to be argued with.

Risks

  • Stage 6 is where this architecture is most easily undermined: if the gate is advisory, it will be skipped during an incident, which is exactly when it matters (Question 1, ADR-03).
  • Stage 4 runs in the compute the pipelines also need. Quality work competes for capacity precisely when things are going wrong (ADR-07).
◀  System Context All views Actors and Journeys  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records02-high-level-architecture