Data Quality Service  ·  View 01 of 20  ·  Context and scope

System Context

Who asks this service questions, what it reads, and what it deliberately does not own.

Editable source SVG draw.io All views
Producers and orchestration Ingestion pipelines orders · rides · payments Orchestrator reads state before run People it serves Data product owner 42 domains Data engineer ~180 Data steward per domain Lakehouse it judges Delta Lake tables 2.4 PB · 12,000 datasets Unity Catalog governance + lineage Databricks Jobs / SQL push-down compute Data Quality Service judge · record · gate Consumers and signalling Analyst / scientist ~2,400 BI and reporting regulatory + exec Feature store pricing · ranking Paging and ticketing Tier 1 breaches owns tiers, SLAs authors assertions overrides versions lineage SQL plans commit events gate check state scorecards known-good incidents Data Quality Service — System Context External / third party Person or role Data store Security / platform synchronous event / async Quarantine tables are omitted here and drawn in views 09 and 13. Out of scope: scheduling pipelines, owning producer contracts, and repairing data — the service judges, records and gates, and never writes a corrected value. v 1.0 · owner Data Platform Architecture · date 2026-09

Decisions

  • The judged estate is read-only to this service. Nothing here writes a corrected value into a dataset; repair happens upstream through the normal ingestion path and is then re-judged (ADR-02).
  • Unity Catalog is the governance spine, not a second source of truth: grants, lineage and table versions are read from it rather than mirrored.
  • The orchestrator is drawn as an actor because it is the largest single reader of state — roughly 9,000 pre-run gate checks a day against ~2,400 human sessions a month.

Assumptions

  • 12,000 registered datasets across 42 domains, 2.4 PB, ~40 TB/day ingest. Stated assumption, sized on a mid-size consumer super-app.
  • ~180 data engineers author rules; ~2,400 analysts and scientists read state monthly.
  • Paging and ticketing already exist as platform services and are integrated, not built.

Out of scope

  • Scheduling or running the pipelines that produce the data.
  • Owning the data contract negotiation with producers — the service imports contracts, it does not arbitrate them.
  • Master data management and record-level reconciliation between systems of record.
  • Quarantine tables are omitted from this view to keep the edges readable; they appear in views 09 and 13.
All views High-Level Architecture  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records01-system-context