Data Quality Service  ·  View 08 of 20  ·  Structure

Integration Surface

Every interface the service commits to, and the one that everything else depends on.

Editable source SVG draw.io All views
Callers and producers Orchestrator pre-run gate check BI + feature store known-good pointer Rules CI Git merge Commit event stream MSK topic Source control totals ledger · OLTP counts Data Quality Service State API ≤ 20 ms p99 Rule + override API owner-scoped Verdict event feed at-least-once Verdict query API history · scorecards Dependencies Unity Catalog read: grants · lineage Databricks Jobs / SQL run: fused scan plans Paging + ticketing Tier 1 only state state rules versions totals Integration Surface — Every Interface the Service Commits To External / third party Interface / broker Queue / topic Security / platform synchronous event / async batch Direction is on the arrows and the interaction is on each dependency's own line; identity is drawn in view 20 and the outbound paging integration in view 18. The State API is the contract the rest of the platform depends on — everything else can degrade, but if that read stops answering, every consumer needs an answer that is not 'assume good'. v 1.0 · owner Data Platform Architecture · date 2026-09

The contract that matters

  • The State API is the platform-wide contract: one cheap question — 'is this dataset safe to read right now' — answered uniformly, with an explicit staleness bound on the answer (ADR-04).
  • Every other interface may degrade. If the State API stops answering, consumers must have a defined behaviour that is not 'assume good' (ADR-03).
  • The verdict event feed is at-least-once and carries no payload values — subscribers get outcomes and pointers, never rows (ADR-14).

Assumptions

  • 4,000 state reads/second sustained, 12,000/second peak. Stated assumption, dominated by the orchestrator's pre-run checks.
  • Source control totals arrive as a batch feed from the ledger and OLTP systems, once per accounting period.

Deliberate omissions

  • Identity is drawn in view 20 and the outbound paging integration in view 18; both are omitted here so the interface fan-out stays legible.
  • Direction is carried by the arrowheads and each dependency's interaction by its own line, rather than by edge labels that would stack at the fan-out point.
◀  Container View All views Storage Zones  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records08-integration-surface