Data Quality Service  ·  View 07 of 20  ·  Structure

Container View

Control, evaluation and verdict planes, and which of them is allowed to touch the judged estate.

Editable source SVG draw.io All views
AWS eu-west-1 · platform account Control plane — EKS, private subnets Rule catalogue API Go · Aurora-backed Scheduler priority + quota Plan compiler DSL → SQL Reconciliation sweeper 10-min cadence Override service expiry enforced Evaluation plane — Databricks workspace Metadata evaluator catalogue reads SQL warehouse pool fused scans Distribution job baseline compare Backfill lane separate pool Verdict + state plane Verdict store Delta · append-only Current-state index DynamoDB global Baseline store Delta · versioned Blast-radius resolver lineage walk Catalogue + debt DB Aurora PostgreSQL Delta Lake on S3 2.4 PB judged estate Unity Catalog grants · lineage Commit event stream MSK Orchestrator reads state commit events work item compiled plan verdicts expiry record Container View — Control, Evaluation and Verdict Planes Application we own Security / platform Data store Queue / topic External / third party event / async synchronous Seven of about thirty edges are drawn; the in-place scan and the sweeper's catalogue poll are omitted for legibility. The edge that matters is absent by design: nothing in the control or verdict plane writes to the judged estate. v 1.0 · owner Data Platform Architecture · date 2026-09

Decisions

  • Evaluation lives inside the Databricks workspace, not in a dedicated validation runtime: computation goes to the data, and no dataset leaves its storage layer to be judged (ADR-07).
  • Two stores, not one, behind the read path: the verdict store is the durable record and the current-state index is a derived projection, rebuildable from it (ADR-04).
  • The backfill lane is a separate warehouse, so re-judging history can never delay judging the present (ADR-10).

Assumptions

  • Aurora PostgreSQL for the catalogue and debt register; DynamoDB global tables for the state index; Delta on S3 for verdicts and baselines. Stated assumptions consistent with the AWS stack.
  • MSK carries both the inbound commit topic and the outbound verdict feed.

Risks

  • The scheduler is a leader-elected singleton. Its failure stops new work being enqueued — recoverable, but the reconciliation sweeper is what makes it recoverable without loss (ADR-09).
  • Sharing the workspace's compute means a warehouse saturated by pipeline work delays Tier 1 verdicts exactly when the platform is unhealthy.
◀  Layered Architecture All views Integration Surface  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records07-container-view