# Data Quality Service

**Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records**

The outage that never gets a status page: a table that is present, fresh-looking and wrong. The job succeeded, the dashboard rendered, the pricing model scored, and nobody was paged — because nothing failed. Four weeks later a merchant disputes an invoice and someone discovers a currency column silently changed units. This is the architecture of the service that judges whether data in a consumer super-app's lakehouse is fit to use, attaches that judgement to the data itself, and decides whether bad data is allowed to circulate: 12,000 datasets, 2.4 PB, 180,000 assertion evaluations a day, and 12,000 state reads a second from the orchestrator tasks that ask, before every run, whether their input is safe. Built from Delta Lake table versions as the subject of every verdict, Unity Catalog as the governance spine, Databricks SQL warehouses for push-down evaluation, DynamoDB for the state index, Aurora for the catalogue and the debt register, and S3 Object Lock for the override trail.

The design rests on one rule: **a verdict binds to an immutable data version, never to the pipeline run that produced it — and the plane that judges data is separate from the plane that circulates it.**

The decisions that carry the design:

- **Bind the judgement to the data, not the run.** A rule written in September can be evaluated against July's version 38,914 and produce a verdict as authoritative as a live one. Key verdicts by run instead and history becomes un-re-judgeable, every new rule starts from today, and the only answer to "how far back does this go" is a forensic investigation (ADR-01).
- **The service judges; it never repairs.** No component writes a corrected value into a dataset. Blocking is non-promotion, so the rejected version stays inspectable and a quality-service outage degrades to "consumers read the last known good version" (ADR-02).
- **The gate is a separate step that reads published state.** Enforcement airtight enough to be owned by the judge would make a careful system a hard dependency of every write in the platform. Promotion stays in the ingestion framework, implemented once, and consumers that stop asking are alerted on (ADR-03).
- **Absence of a verdict is never a pass.** Five states, not two, with coverage rendered alongside every one of them. "Forty assertions passed" and "nobody ever wrote one" must not look the same — that collapse is the defect that produces the silent-corruption case (ADR-05).
- **Computation goes to the data, on declared lanes.** Assertions compile to bounded, fusable plans and run in the engine the pipelines already use; backfill runs on interruptible capacity so re-judging history never delays judging the present (ADR-06, ADR-07).
- **A new rule is run against history before it may enforce.** Automatic 30-day backfill, a would-have-failed report, and a human answering the one question the platform cannot: is this a discovery or a bad rule (ADR-10)?
- **A bad rule cannot take down a pipeline.** Per-assertion circuit breaking, thirty days of probation, and automatic demotion above a 5% false-positive rate — the number that decides whether this platform is enforced or decorative (ADR-12).
- **Quarantine only where a violation is row-attributable**, with a 2% ceiling that escalates to block, because quarantining a quarter of a batch disguises a broken upstream as partial success (ADR-15).
- **A verdict must not become a side channel.** Scoped workload identity, catalogue grants enforced at authoring and execution, samples redacted by default, and an override log the platform's own operators cannot rewrite (ADR-14).
- **Every escape hatch expires and creates debt.** Overrides, suppressions and probations are objects with an owner, a justification and an expiry; unbounded suppression is not supported by the API (ADR-16).

The architecture one-pager (including why the design should still hold up in ten years) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~15 min) and [docs/decision-record.md](docs/decision-record.md) (~70 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 20 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 16 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..d.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only) |
| `ask.md` | The requirement |

## The twenty views

| # | View | What it answers |
|---|---|---|
| 01 | System Context | Who asks this service questions, and the three things it refuses to own |
| 02 | High-Level Architecture | Seven stages from a declared rule to an answer a consumer can act on |
| 03 | Actors and Journeys | Eight actors, including two machines, and what each gets to do |
| 04 | Journey — Wrong for Three Weeks | An analyst discovers a published number was wrong, and what the platform owes them |
| 05 | Journey — Turning a Rule On | Pricing, backfill and probation, from postmortem action item to enforcement |
| 06 | Layered Architecture | Seven layers, and what each of the middle ones is forbidden to know |
| 07 | Container View | Control, evaluation and verdict planes, and which may touch the judged estate |
| 08 | Integration Surface | Every interface committed to, and the one everything else depends on |
| 09 | Storage Zones | Three zones divided by what can be rebuilt and what would be gone for good |
| 10 | Data Model | Twelve entities, and the one foreign key the architecture rests on |
| 11 | Data Flow | From a committed version to an answerable state, and the two arrows that go back |
| 12 | Critical Flow — Commit to Gate | Eighteen messages, including the one where a dependent job decides to skip |
| 13 | Advise, Quarantine, Block | Five outcome classes, what each does to the data, and the way back from each |
| 14 | Retroactive Evaluation | Re-judging history, the corruption window, and scoping the damage |
| 15 | Assertion Lifecycle | The loop that keeps rules credible, and where it can stall |
| 16 | Deployment Architecture | One region that evaluates, a second that can still answer |
| 17 | Rules as Code | From a pull request to an enforcing assertion, and what sends it back |
| 18 | Observability | Signal type by plane, plus the absence checks most platforms lack |
| 19 | Security Zones | Five zones, and why a verdict must not become a side channel |
| 20 | Identity and Override Flow | Who may block, who may let it through, and why no exemption is permanent |

## Rebuilding

```bash
cd architecture/deliverables/use-cases/data-quality-service
bash scripts/build.sh
```

Node 20+ and nothing else. No draw.io Desktop, no browser, no network. The build regenerates
`specs/views.json`, `specs/manifest.json` and `specs/adr.json` from the authoring parts, then
runs generate → validate → route check → SVG → draw.io → HTML → link check, and finally renders
the one-pager and decision record into `diagrams/index.html` and `docs/`.

Edit the parts, never the assembled files.