Data Quality Service  ·  View 11 of 20  ·  Data

Data Flow

From a committed table version to a state a consumer can act on, and the two arrows that go back.

Editable source SVG draw.io All views
Sources Ingestion pipelines ~40 TB/day Source control totals ledger counts Landing Delta table version written, unpromoted Commit event MSK Judging Fused scan N assertions, 1 pass Metadata checks schema · freshness Recording Verdicts immutable rows Quarantined rows + verdict context Deriving Current state per dataset+partition Scorecards domain rollup Debt register aged Serving Gate decision promote / hold Verdict event feed subscribers Consumers BI + reporting Feature store Orchestrator reconcile derive promote Data Flow — From a Committed Version to an Answerable State External / third party Data store Queue / topic Application we own Decision point batch event / async synchronous Two arrows leave this diagram going backwards: the reconcile feed and the promote decision. Everything else moves one way, which is the property that makes a verdict re-derivable. v 1.0 · owner Data Platform Architecture · date 2026-09

Why it is one-way

  • Everything between landing and serving moves forward, which is the property that makes a verdict re-derivable from stored inputs at any later date (ADR-01).
  • Only two arrows go backwards: the reconcile feed from the source-of-record control totals, and the gate's promote decision. Both are drawn because both are where the design could be misread.
  • Judging happens against the written-but-unpromoted version. Blocking is therefore non-promotion, not a rollback of data that already circulated (ADR-02).

Numbers

  • ~40 TB/day ingest across 12,000 datasets; verdict published within 90 s p95 of a Tier 1 commit.
  • Fused scan: N assertions in one pass over a partition, with the achieved fusion ratio reported per dataset (ADR-06).

Risks

  • The derive step from verdicts to state is asynchronous. Its lag is the window in which a consumer can read 'good' after a block was issued — which is why the staleness bound is part of the API contract (ADR-04).
◀  Data Model All views Critical Flow — Commit to Gate  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records11-verdict-data-flow