Data Quality Service  ·  View 10 of 20  ·  Data

Data Model

Twelve entities, and the one foreign key the whole architecture rests on.

Editable source SVG draw.io All views
dataset dataset_id PK catalogue_fqn UQ tier 1|2|3 owner_team staleness_sla_min dataset_version version_id PK dataset_id FK commit_ts watermark_ts row_count promoted bool assertion assertion_id PK dataset_id FK type action advise|quar|block owner_team status assertion_version assertion_ver_id PK assertion_id FK expr_hash published_ts probation_until verdict verdict_id PK version_id FK assertion_ver_id FK outcome retroactive bool sample_fraction evaluated_ts dataset_state dataset_id PK partition_key PK state last_known_good FK as_of_ts baseline baseline_id PK assertion_id FK trained_from trained_to stats_json quarantine_batch batch_id PK verdict_id FK row_count pct_of_batch expires_ts corruption_window window_id PK dataset_id FK first_failing_ver first_passing_ver discovered_ts override override_id PK verdict_id FK granted_by justification expires_ts debt_item debt_id PK kind subject_id owner_team opened_ts age_days suspect_mark mark_id PK window_id FK downstream_dataset_id FK hops cleared_ts null 1 : N 1 : N 1 : N 1 : N 1 : N 1 : N 0 : 1 0 : N 0 : 1 1 : N 1 : 1 Data Model — The Verdict and What It Points At verdict.version_id is the load-bearing key: it binds a judgement to an immutable data version rather than to a pipeline run, which is what makes retroactive evaluation and corruption_window expressible at all. v 1.0 · owner Data Platform Architecture · date 2026-09

The load-bearing key

  • verdict.version_id binds a judgement to an immutable dataset version rather than to a pipeline run. Without it, corruption_window and retroactive verdicts are not expressible at all (ADR-01).
  • verdict also carries assertion_ver_id, so a verdict names both what was judged and the exact rule that judged it — which is what makes a revised rule a new verdict rather than an edit.
  • override and debt_item are one-to-one on purpose: granting an escape hatch creates a debt record in the same transaction, so there is no way to have the first without the second (ADR-16).

Assumptions

  • dataset_state is keyed by dataset and partition, not by dataset alone — partition-level state is what makes a late partition distinguishable from a broken table.
  • verdict rows are immutable; a revised judgement supersedes rather than updates, and both remain readable.

Risks

  • suspect_mark.cleared_ts is nullable and manual. An uncleared backlog of marks is the failure mode that makes the mark meaningless (ADR-11).
  • sample_fraction on verdict exists so a sampled pass is never read as a full-scan pass. If a consumer ignores it, the field is decoration.
◀  Storage Zones All views Data Flow  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records10-data-model