Data Quality Service  ·  View 12 of 20  ·  Runtime

Critical Flow — Commit to Gate

Eighteen messages from a written table version to an orchestrator deciding whether to start.

Editable source SVG draw.io All views
Ingestion pipeline Delta table Scheduler Evaluator Verdict store State index Orchestrator 1. commit version 41207 (unpromoted) 2. commit event 3. mark unknown 4. fuse assertions for this partition 5. metadata set first 6. schema + freshness verdicts 7. schema passed — proceed to scan 8. fused scan plan, budget attached 9. single pass over the partition 10. aggregate + row results 11. verdicts bound to v41207 12. derive state: blocked 13. withhold promotion 14. is my input safe to read? 15. blocked · last known good v41198 16. skip dependent run, do not cascade 17. budget exhausted on a Tier 2 dataset 18. degraded, not pass Critical Flow — Table Version Committed to Gate Answered The last two messages are the alternate path: a verdict that could not be fully reached is recorded as degraded. Nothing in this flow ever writes 'pass' because evaluation did not finish. v 1.0 · owner Data Platform Architecture · date 2026-09

The three that carry it

  • Message 3 marks the new version 'unknown' before anything is evaluated. Absence of a verdict is never a pass, so the window between commit and verdict is explicitly not-yet-judged (ADR-05).
  • Message 7 short-circuits: if a schema assertion fails, row predicates against the changed schema produce meaningless verdicts and are not run (ADR-06).
  • Message 16 is the payoff of the gate being readable before a run: the dependent job skips rather than reading stale data and succeeding, which is how one bad version fails to become a cascade (ADR-03).

The alternate path

  • The last two messages are the budget-exhaustion case: a Tier 2 dataset whose assertion set could not finish is recorded as 'degraded', never as 'pass' (ADR-08).
  • A Tier 1 dataset in unknown or degraded state is not promoted — fail-safe, not fail-open.

Numbers

  • Verdict published within 90 s p95 / 180 s p99 of a Tier 1 commit; gate read answered in ≤ 20 ms p99.
  • Version numbers in the diagram are illustrative. Stated assumption.
◀  Data Flow All views Advise, Quarantine, Block — and Unknown  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records12-critical-flow-commit-to-gate