Architecture One-Pager

Data Quality Service · Solution Architecture v1.0 · Databricks Lakehouse on AWS · Data Platform Architecture · 2026-09 · 20 views · 16 architecture decision records

A verdict binds to an immutable data version, never to the pipeline run that produced it — and the plane that judges data is separate from the plane that circulates it.

Every consumer product people use daily runs a lakehouse behind it, and every one of those lakehouses has the same recurring outage that never gets a status page: a table that is present, fresh-looking and wrong. The job succeeded. The dashboard rendered. The pricing model scored. Nobody was paged, because nothing failed. Four weeks later a merchant disputes an invoice and someone discovers that a currency column silently switched units, or that a restaurant_is_open flag has been true for a chain that shut in March. At that point the useful question is not 'how do we fix it' — the fix is an hour's work upstream — but 'how much of the last four weeks do we have to disbelieve, and who else read it'. Almost no data quality tooling can answer that, because almost all of it records that last night's run passed rather than that a specific version of a specific table was judged and found sound.

Assertions are versioned, owned, priced and compiled to bounded query plans, then pushed down into the lakehouse engine so no dataset ever leaves its storage layer to be judged. Evaluation is triggered by a table-version commit, by a schedule, and by a reconciliation sweep that exists because the event stream will lose something eventually. Every evaluation writes an immutable verdict keyed by the data version and the assertion version that produced it — never by the run — and a derived state index answers the platform's one hot question, 'is this dataset safe to read right now', in under twenty milliseconds with an explicit staleness bound. Enforcement is a separate step that reads that state: an offending row-level batch is quarantined under the source's own classification, a dataset-level failure withholds promotion of the new version so consumers keep reading the last known good one, and nothing anywhere writes a corrected value back into a dataset. Because verdicts are bound to versions, a rule written today can be evaluated against any past version, which is what turns the four-week question into a corruption window, a blast-radius list, and a set of suspect marks somebody has to clear.

What it is, and what it is not

• A verdict keyed by the data version and the assertion version — not a record that last night's pipeline run passed, which cannot be re-derived, cannot be re-judged by a rule written later, and gives a forensic investigation as the only answer to 'how far back does this go'.
• Blocking implemented as non-promotion of a written version — not deleting, rolling back or mutating the data, which destroys the evidence and makes the quality service a writer to the estate it is meant to judge.
• 'Unknown' and 'no coverage' rendered as their own states — not a green tick that means either 'forty assertions passed' or 'nobody ever wrote one', which is the defect that lets a silently corrupt table look healthy for a month.
• Quarantine restricted to row-attributable violations, with a ceiling that escalates to block — not quarantine as a general-purpose safety valve, where diverting a quarter of a batch disguises a broken upstream as partial success.
• A cost estimate posted before an assertion is published, and a per-dataset budget at run time — not unbounded validation queries discovered in next month's bill, which is how quality coverage gets cut as a cost measure.
• Automatic demotion of an assertion above a measured false-positive rate — not a growing set of alerts everyone has learned to ignore, which is the normal end state of a data quality programme.
• Overrides and suppressions as expiring objects that create a debt item — not configuration flags, where the exception outlives the incident and nobody can list what is currently unguarded.
• Evaluation pushed down into the engine the pipelines already use — not a separate validation runtime that copies data out to check it, duplicating cost, storage governance and the credential path into every dataset.

The decisions that are the architecture

1. Verdicts bind to data versions, not to runs (ADR-01) — The single decision the whole architecture rests on. It makes history re-judgeable, lets a rule written in September condemn a window in July, and turns 'how far back does this go' into a query.
2. The service judges; it never repairs (ADR-02) — No component writes a corrected value into a dataset. Blocking is non-promotion, so the rejected version stays inspectable and a service outage degrades to 'consumers read the last known good'.
3. The gate is a separate step that reads state (ADR-03) — The quality service publishes a claim about a version; promotion is a distinct act that consults it. That keeps the service off every write path without making enforcement a matter of goodwill.
4. Current state is a derived projection with a published staleness bound (ADR-04) — Four thousand reads a second against a far lower verdict rate. The index is rebuildable from the verdict store, which is why its recovery objective is a rebuild rather than a restore.
5. Absence of a verdict is never a pass (ADR-05) — 'Unknown' and 'no coverage' are first-class states, and a Tier 1 dataset in either is not promoted. Fail-safe, not fail-open.
6. Computation goes to the data, on declared priority lanes (ADR-07) — Assertions compile to engine-native plans and fuse into one pass per partition. Backfill runs in its own lane so re-judging history never delays judging the present.
7. Retroactive evaluation with a mandatory human judgement (ADR-10) — A new rule is backfilled over a 30-day lookback and the result reported before it may enforce, because a rule that condemns 18 of the last 30 versions is as likely to be a bad rule as a discovery.
8. A bad rule cannot take down a pipeline (ADR-12) — Per-assertion circuit breaking, probation for new rules, and automatic demotion above a 5% false-positive rate — structural, not operational courtesies.
9. Every escape hatch expires and creates debt (ADR-16) — Overrides and suppressions are objects with an owner, a justification and an expiry, written to an immutable log before the state changes. Unbounded suppression is unsupported.

Why this should still be right in ten years

A data quality service outlives the warehouse it was built against, because the judgements it recorded are what a future team uses to decide whether to trust a number from today. These are the properties that should survive a change of engine, of cloud, and of the people who built it.

• The version binding depends on no technology. ADR-01 needs one thing from the storage layer: an immutable, addressable version of a dataset. Delta gives it, Iceberg gives it, Hudi gives it, and a partitioned object layout with a manifest gives it. What would break the decision is not a migration but a future team recording verdicts against a job id because it was easier to join — which is why the version key is drawn in the data model rather than left to an implementation note.
• 'Unknown is not pass' is the claim that ages best. Every threshold in this design will be renegotiated: the 2% ceiling, the 5% false-positive rate, the 15-minute Tier 1 SLA, the 2.5% cost cap. None of them is load-bearing in the way ADR-05 is. A platform that renders 'nobody checked' and 'forty checks passed' identically will produce the silent-corruption case again no matter how good its rules are, and that is true of any tooling in any decade.
• The judging/circulating separation is what keeps the service safe to depend on. Because the quality service only publishes claims, its own failure has a defined, benign meaning: consumers read the last known good version. The moment a future team puts promotion inside the quality service to make enforcement airtight — a reasonable-sounding change — a quality service outage becomes a platform-wide write outage. ADR-03 states the boundary explicitly so that trade is made deliberately if it is ever made.
• Expiring exceptions are the only mechanism that survives organisational pressure. Quality debt does not accumulate because people are careless; it accumulates because an exception granted during an incident has no natural end. ADR-16 makes the end structural. Of everything here, this is the decision most likely to be quietly weakened — by a 'permanent exemption' field added for a good reason — and the one whose loss would be hardest to detect.
• The false-positive rate is what decides whether any of this is enforced. The uncomfortable truth about data quality platforms is that they fail socially, not technically: the alerts become noise and the gate becomes advisory in practice while remaining mandatory on paper. Measuring the false-positive rate per assertion and acting on it automatically is the only defence that does not depend on someone's attention, and it will still be the answer when the assertions are generated by a model rather than typed by an engineer.

Non-functional targets

Every number here is a stated assumption from the requirement, chosen to be argued with. The right-hand column names the view where the mechanism that meets it is drawn.

Scope

In scope

• Versioned assertion authoring in a rule DSL held in version control, with tier and verdict action declared at authoring time.
• Push-down evaluation triggered by table-version commit, by schedule, and by a reconciliation sweep, with assertions fused per dataset scan.
• An immutable verdict store keyed by data version and assertion version, and a derived current-state index with a published staleness bound.
• Freshness SLAs per dataset tier, distinguishing arrival staleness from event-time staleness.
• Three verdict actions — advise, quarantine, block — with blocking implemented as non-promotion and quarantine restricted to row-attributable violations.
• Retroactive evaluation, automatic backfill of new assertions, corruption-window computation and one-hop suspect propagation.
• Attributed, expiring overrides and suppressions, an immutable audit log, and a quality debt register that ages visibly.
• Per-dataset and per-domain scorecards, cost attribution, and the detection-quality metrics that say whether the practice is working.

Explicitly out of scope

• Scheduling or running the pipelines that produce the data — the service consumes their commits and never orchestrates them.
• Repairing data. No component writes a corrected value; repair is upstream and is then re-judged.
• Owning or arbitrating data contracts with producers. Contracts are imported as a source of assertions, not negotiated here.
• Master data management, entity resolution, and record-level reconciliation between systems of record.
• Full transitive suspect propagation and column-level lineage — deferred to Phase 3 pending lineage precision (ADR-11).
• Continuously-written and streaming table evaluation at checkpoint granularity — deferred to Phase 3, and the open question the MVP does not answer.
• Automated re-derivation of downstream datasets once a corruption window closes.

What a four-week prototype should prove

Three of this architecture's claims are cheap to test and expensive to be wrong about. A prototype that proves these can be built on; one that skips them has demonstrated running a SQL query against a table, which is the easy half.

1. Retroactive equivalence: take a rule, evaluate it live against ten consecutive versions of a real Tier 1 table, then evaluate the same rule version against the same ten versions as a backfill a week later, and show the verdicts are byte-identical. If time travel and the live path can disagree, ADR-01 is a slogan rather than a mechanism.
2. Fusion under realistic assertion counts: forty assertions of mixed type on one partitioned fact table, and the achieved fusion ratio and wall-clock cost measured against evaluating them one at a time. The cost model in ADR-08 assumes the ratio is high; if it is not, the 2.5% ceiling forces a coverage decision nobody has made.
3. The gate under a control-plane outage: kill the control plane and show that the orchestrator's pre-run check still answers from the state index, with a staleness value the consumer can act on, and that a Tier 1 dataset in unknown state is not promoted. ADR-03, ADR-04 and ADR-05 are one claim from the consumer's side, and this is the test of it.

• Drop a commit event for a Tier 1 table and confirm the reconciliation sweeper enqueues the unjudged version within one cycle, that the version reads 'unknown' rather than 'good' in the meantime, and that it is not promoted.
• Kill the control plane mid-evaluation and confirm the orchestrator's pre-run check still answers from the state index, with a staleness value attached, and that a consumer past the published bound degrades to unknown rather than reusing its last answer.
• Publish an assertion whose plan the compiler cannot bound and confirm it is refused at merge rather than discovered at run time; then publish one that exceeds its dataset budget at run time and confirm the verdict is 'degraded' with the metadata assertions still evaluated.
• Exhaust an assertion's execution repeatedly — a deliberately poisonous predicate — and confirm the circuit breaks for that assertion alone while its thirty-nine siblings on the same table continue to produce verdicts.
• Grant an override on a blocked version, let it expire, and confirm the state reverts without anyone acting, the debt item ages rather than closing, and the audit entry was written before the state changed.

Open risks, carried rather than hidden

The reasoning behind every component and technology choice is in the Architecture Decision Record: 16 records across 6 areas, each with the alternatives that lost and what the choice costs.