Data Quality Service  ·  View 16 of 20  ·  Operations

Deployment Architecture

One region that evaluates, a second that can still answer, and an estate that belongs to neither.

Editable source SVG draw.io All views
AWS eu-west-1 · primary Control plane — EKS across three AZs EKS node groups 3 AZ · 12 nodes Catalogue + state APIs HPA · 6–40 pods Scheduler leader-elected Aurora PostgreSQL multi-AZ writer Evaluation plane — Databricks workspace SQL warehouses live lane · autoscaled Backfill warehouse spot · low priority Job clusters distribution + reconcile Regional data S3 — verdicts, baselines, quarantine versioned · Object Lock on audit MSK commit + verdict topics AWS eu-central-1 · read replica of state State read path only DynamoDB global table replica · ≤ 2 s lag State API replica reads only Cold recovery S3 cross-region replication verdicts + audit log DynamoDB global table · primary current-state index Delta Lake on S3 — the judged estate not owned by this service Unity Catalog account-level state writes replication CRR Deployment — Two Regions, One Judged Estate Security / platform Application we own Data store Queue / topic synchronous event / async batch Four of the region's edges are drawn; the in-place scan of the judged estate is omitted here and drawn in views 07 and 11. The second region carries the state read path and the durable record, not the ability to evaluate: a regional loss still answers 'is this safe to read' within 2 s of lag, but produces no new verdicts. v 1.0 · owner Data Platform Architecture · date 2026-09

Decisions

  • The second region carries the state read path and the durable record, not the ability to evaluate. A regional loss means consumers can still ask 'is this safe to read' and get an answer bounded by 2 s of lag; it does not mean new verdicts appear (ADR-04).
  • The backfill warehouse runs on spot capacity at low priority — the one place in the design where interruption is genuinely acceptable (ADR-10).
  • Object Lock on the override audit log, not on the verdict store: verdicts are append-only by convention and rebuilt if lost, but the override log is the artefact an auditor asks for (ADR-14).

Numbers

  • Control plane ≥ 99.5% monthly, state read path ≥ 99.95%, evaluation plane ≥ 99.0%. Stated assumptions.
  • State index replication lag ≤ 2 s; control plane RPO 5 min / RTO 30 min.
  • Total evaluation cost ceiling 2.5% of platform compute spend (ADR-08).

Deliberate omissions

  • The in-place scan of the judged estate is omitted here and drawn in views 07 and 11.
  • Networking detail — VPC endpoints, subnet CIDRs, the Databricks private link path — is a build-time concern and is not architecture at this level.
◀  Assertion Lifecycle All views Rules as Code  ▶
Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records16-deployment