Architecture Views

When the service records that a dataset passed or failed, what exactly is the subject of that claim — the run that produced the data, or the data itself?

Context

Every data quality tool in common use records run outcomes, because that is where the check executes: the pipeline calls the validator, the validator returns pass or fail, and the result is written against the run id. It reads perfectly well and it is almost useless six months later. A run-keyed verdict cannot be re-derived, because the run is gone. It cannot be revised, because a rule written later has nothing to attach to. And it cannot answer the only question that matters during a real incident — 'which weeks of this table should we disbelieve' — because the verdicts describe executions rather than data. Meanwhile the lakehouse already provides exactly the primitive needed: every commit produces an immutable, addressable version of the table, readable long after the job that wrote it has been deleted.

Decision

Every verdict is keyed by the pair (dataset version, assertion version) and is immutable once written. A verdict names the exact data it judged and the exact rule that judged it. Re-evaluating the same pair is idempotent and produces no second row. A revised judgement — a corrected rule, a re-run after an engine upgrade — is a new verdict that supersedes an earlier one, and both remain readable. The run that triggered the evaluation is recorded as metadata on the verdict, never as its key.

How it is realised on Databricks and AWS

Delta Lake table versions are the subject: the evaluator reads a specific version through time travel, and the verdict row carries `version_id` as a foreign key alongside `assertion_ver_id`. Verdicts land in an append-only Delta table on S3 partitioned by evaluation date, with the (version, assertion version) pair unique. Unity Catalog supplies the version history the reconciliation sweeper diffs against. The same evaluator code path serves live and retroactive evaluation, which is what makes the two produce identical verdicts — and is the property the four-week prototype is asked to demonstrate.

Options weighed

• ChosenKey the verdict by (data version, assertion version): Makes history re-judgeable and corruption windows computable. Costs a dependency on the table format's version primitive, and a verdict store that grows with versions × assertions rather than with runs.
• RejectedKey the verdict by pipeline run: The natural choice, and the one that makes the silent-corruption case unanswerable. A rule written today can never say anything about yesterday, so every quality programme starts from the day its rules were written.
• RejectedWrite verdicts as table properties and commit metadata inside the lakehouse itself: Attractive: a table would always carry its own verdict and could never be separated from it. Rejected because a dropped or rolled-back table takes its quality history with it — precisely when the history matters most — and because cross-dataset verdict queries for scorecards become expensive scans over thousands of table histories.
• RejectedKey by (dataset, partition, evaluation timestamp): Close enough to work most of the time, which makes it dangerous. Two versions of the same partition written minutes apart get conflated, and a re-evaluation is indistinguishable from a new judgement.

Consequences

What it buys

• Any past state of any table can be re-judged, so a rule written today can say which of last month's versions to disbelieve.
• A corruption window is computable as an interval between two versions rather than reconstructed by hand.
• The ML engineer's question — 'was the snapshot I trained on sound' — has an answer years later.
• Re-evaluation is idempotent, so a re-run after an engine upgrade cannot silently double-count or contradict itself.

What it costs

• A verdict store sized by versions × assertions rather than by runs — roughly 180,000 rows a day at the assumed scale.
• A hard dependency on the table format keeping old versions available for the full 25-month retention window.
• A vacuum policy that must never delete a version a retained verdict still references, which is a coupling somebody has to own.

Choose differently when

If the storage layer stopped providing addressable immutable versions — a migration to a format without time travel, or a retention policy that vacuums aggressively enough to make the lookback window meaningless — the decision would have to change, and with it the whole retroactive capability. The honest signal to watch is the gap between the verdict retention window (25 months) and the table history actually retained; if the second is shorter, the first is a promise the platform cannot keep.

Why it holds up over time

This is the decision most likely to still be right in ten years, because it asks almost nothing of the technology: one immutable, addressable version of a dataset, which Delta, Iceberg, Hudi and even a manifest-plus-objects layout all provide. What would break it is not a migration but convenience — a future team joining verdicts to a job id because the pipeline already had one to hand. That is why the version key appears in the data model view rather than in an implementation note.

When the service finds bad data, may it change anything — fix a value, drop a row, roll back a version — or may it only decline to let the data circulate?

Context

The tempting scope creep is repair. The service already knows the rule, has the offending rows in hand, and could coerce the currency, null the impossible timestamp, or roll the table back to the last good version. Every one of those makes the quality service a writer to the estate it judges, which has three consequences that arrive later and hurt more than the original defect: the data no longer matches what the producer sent, so the producer cannot reproduce it; the evidence of what went wrong is destroyed by the fix; and the service becomes a component whose bug corrupts production data rather than one whose bug delays a verdict. Rollback is the subtlest version of the same mistake — it looks like restoring a good state, and it removes the version an auditor would need to see.

Decision

The service has no write path into any judged dataset. A failing dataset-level assertion results in the new version not being promoted to the consumable state; consumers continue reading the last known good version, and the failed version remains in the table history, inspectable. A failing row-level assertion results in offending rows being diverted to a quarantine dataset and the remainder promoted — a write to quarantine, never a modification of the source. Repair happens upstream, through the producer's normal ingestion path, and the corrected data is judged again from scratch.

How it is realised on Databricks and AWS

Promotion is a property change, not a data movement: the consumable pointer for a dataset advances to a version only when its state permits. A blocked version stays in Delta history and is reachable by version number for anyone inspecting it. Quarantine is a separate Delta table under the source dataset's classification, holding the diverted rows plus the verdict context. Re-admission of corrected rows goes back through the ingestion pipeline, so quarantined rows enter the target exactly as any other rows do, and are judged accordingly.

Options weighed

• ChosenJudge only; blocking is non-promotion: The failed version stays inspectable, the producer's output is never altered, and the service's blast radius on a bug is a delayed or wrong verdict rather than corrupted data.
• RejectedRoll the table back to the last good version: Looks equivalent and is not: it destroys the version an investigation needs, it fights the producer's next write, and on a table written every few minutes it becomes a race the quality service will sometimes lose.
• RejectedCoerce or repair known-fixable violations in place: Right for a data preparation tool, wrong for a judge. The repaired data no longer matches the source of record, and the defect stops being visible to the team that could actually fix its cause.
• RejectedQuarantine by deleting offending rows from the target: A delete is a write. It also makes the remainder's row count inconsistent with the producer's own count, which then fails a reconciliation assertion — the platform manufacturing its own defect.

Consequences

What it buys

• The service's failure modes are 'late verdict' and 'wrong verdict', never 'lost data' — which is what makes it safe to depend on.
• Every rejected version survives and stays inspectable, so an investigation has evidence rather than a description.
• The producer's output is never altered, so a defect stays visible to the team that can fix its cause.

What it costs

• Consumers can be served stale-but-good data for as long as the producer takes to fix the cause, which on a Tier 1 dataset is felt immediately.
• That cost is the whole reason the override mechanism in ADR-16 exists, and therefore the reason quality debt exists at all.
• Re-admission through the ingestion path is slower than writing corrected rows directly.

Choose differently when

If the platform ever genuinely needs automated remediation — a deterministic, reversible, well-understood coercion applied at scale — it should be a separate service with its own identity, its own audit trail, and its own verdicts produced by this one. Folding it into the judge would be the change to resist, not the capability itself.

Why it holds up over time

The separation of judging from repairing is a boundary, not a technique, and boundaries age well. The pressure on it will come from a specific, reasonable request — 'just trim the whitespace, it fails validation every night' — and the right answer will still be that the producer trims the whitespace and the judge records that they did.

What does the platform say about a dataset that has not been evaluated, could not be evaluated, or has no assertions at all?

Context

This is the defect that produces the silent-corruption case, and it is almost always an interface decision rather than an engineering one. A dataset with forty passing assertions and a dataset nobody has ever written a rule for both render as a green tick, because the underlying model has two values and reality has at least five. The same collapse happens in the failure path: an evaluation that timed out, a table whose metadata could not be read, a freshness assertion that has not run for three cadences — each of these is an absence of information, and each is routinely displayed as health. The analyst in view 04 publishes a wrong number for three weeks not because a check failed but because the absence of a check looked identical to a passing one.

Decision

The state model has five values — `good`, `degraded`, `suspect`, `blocked`, `unknown` — and coverage is a separate, always-rendered dimension. An evaluation that cannot complete produces `degraded`; one that cannot start or cannot read its target produces `unknown`; a dataset with no assertions is reported as zero coverage and never as passing. A Tier 1 dataset in `unknown` state is not promoted. A freshness assertion that has not been evaluated within twice its cadence is treated as breached rather than as silent.

How it is realised on Databricks and AWS

The state index carries the state and a coverage summary in the same record, so no caller can read one without the other. The scheduler marks a newly committed version `unknown` before evaluation begins, so the window between commit and verdict is explicitly not-yet-judged rather than implicitly fine. Absence checks in view 18 — no work enqueued, no verdict in two cadences, climbing unknown count, datasets with zero assertions — are the signals that make this decision operational rather than aspirational.

Options weighed

• ChosenFive states plus an always-rendered coverage dimension: Makes 'nobody checked' impossible to mistake for 'checks passed', and makes fail-safe behaviour on Tier 1 expressible. Costs consumer complexity: five states is more than anyone wants to handle.
• RejectedPass/fail, with coverage on a separate dashboard: The normal design, and the one that reproduces the original defect. Coverage on a different screen is coverage nobody reads at the moment of the decision.
• RejectedTreat unknown as fail: Safe and unworkable: at 12,000 datasets it blocks the platform every time the evaluation plane has a bad hour. It also loses the distinction between 'we judged this and it is wrong' and 'we have not judged it', which is the distinction an incident turns on.
• RejectedTreat unknown as pass, with alerting: What most platforms do by accident. The alert fires into a channel, the state says good, and the consumer believes the state.

Consequences

What it buys

• An unmonitored dataset looks unmonitored — the single property that prevents the silent-corruption case.
• Fail-safe behaviour on Tier 1 becomes expressible, because 'we have not judged this' is a state the gate can act on.
• Absence checks have something to assert against: a climbing unknown count is a signal rather than a gap.

What it costs

• Every consumer has to handle five states rather than two, which needs a client library that makes the safe default easy.
• Real friction on Tier 1: a dataset whose evaluation plane had a bad hour is not promoted, and someone is paged about a delay rather than a defect.
• The console has no two-valued summary to fall back on, which will be argued about by whoever wants a green badge.

Choose differently when

Nothing short of a different problem would flip this. The decision to keep watching is whether five states is too many in practice: if consumers collapse them into two in their own code, the platform has the model right and the client library wrong.

Why it holds up over time

Of every decision here, this is the one most independent of technology and most likely to be quietly eroded — by a well-meaning change that makes the console's summary badge green when there is nothing to report. The defence is that the state model itself has no two-valued projection to fall back on.

Does the quality service itself promote table versions — owning the final step of every write so nothing can circulate unjudged — or does it publish a state that each pipeline and consumer is expected to consult?

Context

This is the sharpest trade-off in the design, and both answers are defensible. If the service owns promotion, enforcement is airtight: there is no path to consumable data that does not pass through the judge. It also makes the quality service a hard availability dependency of every write in the platform, so an outage in a system whose job is to be careful becomes an outage in the systems whose job is to deliver. If the service only publishes state, it stays off every critical path and can be as slow or as unavailable as its own SLO allows — but enforcement becomes a matter of each consumer choosing to ask, and a gate that can be skipped will be skipped during exactly the incident where it matters. The naive framing treats this as a governance question. It is an availability question.

Decision

The quality service publishes claims about versions; promotion is a distinct step, owned by the platform's write path, that consults the published state before advancing the consumable pointer. The service does not perform promotion. To keep this from decaying into advisory enforcement, three things are structural: promotion is implemented once, in the shared ingestion framework, rather than by each pipeline; a Tier 1 dataset in unknown or blocked state cannot be promoted by that framework; and gate reads are themselves instrumented, so a consumer that used to ask and has stopped asking raises an alert.

How it is realised on Databricks and AWS

The shared ingestion framework calls the State API before advancing a dataset's consumable pointer, and the orchestrator calls it before starting any dependent job. The state index is a DynamoDB global table read directly by both, so the read path does not traverse the control plane. The 'no gate reads from a consumer' absence check in view 18 is what makes the arrangement observable rather than assumed.

Options weighed

• ChosenPublish state; promotion is a separate, centrally implemented step: Keeps the judge off the write path and makes its own availability target modest. The enforcement gap is closed by centralising promotion once and by alerting on consumers that stop asking, rather than by owning the write.
• RejectedThe quality service owns promotion: Airtight enforcement at the cost of making a quality-service outage a platform-wide write outage. Right for an organisation where regulatory exposure dominates and write latency does not — a clearing house, not a super-app.
• RejectedEach pipeline implements its own gate check: The version of 'publish state' that actually fails: forty teams implement the check forty ways, three of them invert a boolean, and nobody can say which datasets are gated.
• RejectedEnforce at the query layer instead of at promotion: Appealing — refuse to serve bad data rather than refuse to publish it — but it puts the check on every read at four thousand reads a second, and gives no answer for the consumer reading the table directly from object storage.

Consequences

What it buys

• A quality-service outage means 'no new verdicts', not 'no writes', so the control plane can be deployed and restarted freely.
• The state index is readable independently of the control plane, so consumers keep getting answers during an outage.
• Promotion implemented once in the shared framework means the gate's behaviour is one piece of code rather than forty.

What it costs

• Enforcement depends on the ingestion framework being the only promotion path, which is an organisational property as much as a technical one.
• The day a team publishes a consumable table outside the framework, that dataset is ungated and the platform will not know unless coverage is measured against the catalogue.
• 'No gate reads from a consumer that used to read' has to be an alert, because the weakness is silent by construction.

Choose differently when

Two signals would flip it: a repeated incident where data circulated because a promotion path bypassed the framework, or a regulatory requirement that makes 'we published the state and they did not read it' an unacceptable answer. Either would justify moving promotion inside the service and accepting the availability coupling — which is why the state index is already readable independently of the control plane.

Why it holds up over time

The boundary is the durable part; where the gate physically sits is not. Stating them separately is deliberate, so that a future team can move the gate without accidentally also merging the judging and circulating planes — the change that would make a quality outage a platform outage.

Where does the answer to 'is this dataset safe to read right now' live, and is a consumer allowed to act on an answer that is slightly out of date?

Context

The read/write ratio decides this. Verdicts are written a couple of times a second at the assumed scale; state is read four thousand times a second, mostly by orchestrator tasks asking about inputs before they start. Serving those reads from the verdict store means an analytical store on a latency-sensitive hot path, which it will do badly. Serving them from a strongly consistent transactional store means the platform's highest-volume read path shares a failure domain with the control plane. The uncomfortable part is that any projection can be stale, and during the seconds after a block is issued a consumer can read 'good' for a version that has just been condemned. Pretending otherwise is the actual risk; consumers will build on whatever consistency they observe rather than whatever the documentation claims.

Decision

The current-state index is a derived projection of the verdict store, keyed by dataset and partition, holding the state and the last-known-good version pointer. Every response carries an `as_of` timestamp and therefore an explicit staleness, and the API publishes a bound on it. Consumers are required to treat a stale answer past the bound as `unknown` rather than as the last value seen. The index is never written to directly: it is only ever derived, which is what makes rebuilding it a routine operation rather than a recovery event.

How it is realised on Databricks and AWS

Verdicts land in Delta; a streaming derivation writes the state index into a DynamoDB global table, replicated to a second region with a target lag of two seconds. The State API reads DynamoDB directly and does not traverse the control plane, so a control-plane outage leaves consumers able to get an answer. The index's recovery objective is deliberately asymmetric — RPO 15 minutes, RTO 10 minutes — because its recovery is a rebuild from verdicts rather than a restore from a backup.

Options weighed

• ChosenDerived key-value projection with a published staleness bound: Serves the read volume cheaply, survives a control-plane outage, and rebuilds rather than restores. Costs a window in which a consumer can act on a superseded answer, which is why the bound is part of the contract.
• RejectedServe state from the verdict store directly: One store, no derivation, no staleness — and an analytical store answering twelve thousand reads a second at peak. It would also couple every consumer to the store whose retention and compaction policies change for analytical reasons.
• RejectedA strongly consistent transactional store on the hot path: Removes staleness and adds a hard dependency: the platform's highest-volume read now shares a failure domain with the control plane's database, and cross-region reads become a latency problem rather than a replication one.
• RejectedMaterialise state into the governance catalogue as table properties: Genuinely attractive, because consumers already read the catalogue and the state would travel with the table. Rejected on write amplification and on the catalogue's own rate limits at this commit volume, but worth revisiting if the catalogue gains a high-throughput property API.

Consequences

What it buys

• A read path that absorbs 12,000 reads a second at peak without touching an analytical store.
• Regional redundancy for the one question consumers must always be able to ask.
• Recovery by rebuild rather than restore, which is why the index carries a 10-minute RTO against a 15-minute RPO.

What it costs

• An eventual-consistency window in which a consumer can read 'good' for a version that has just been blocked.
• That window has to be honest: a published staleness bound, a stale-index alert at 60 seconds, and consumers required to degrade to unknown past the bound.
• Two stores to reason about, and a derivation whose lag is a first-class metric rather than an implementation detail.

Choose differently when

If a class of consumer appeared for which any staleness is unacceptable — a real-time financial control reading a Tier 1 aggregate — the answer is a strongly consistent read path for that narrow case, not the removal of the projection for everyone. If the catalogue gains a property API that can absorb the write volume, option four becomes the better answer because it puts the state where consumers already look.

Why it holds up over time

Separating the durable record from the serving projection is one of the few patterns that has survived every generation of data infrastructure, and the reason is the same each time: the two have different read shapes, different retention needs and different recovery objectives. What will change is which technology holds the projection.

If the platform learns about new table versions from an event stream, what happens to a version whose event was never delivered?

Context

Event-driven evaluation is the obvious design and the right one for latency: a commit notice arrives, work is enqueued, a verdict appears in ninety seconds. The failure mode is quiet. An event lost to a consumer group rebalance, a topic retention edge, a producer that wrote without emitting, a bug in the notice path — any of these leaves a version that was never judged, and under ADR-05 that version is `unknown` rather than passing, which is safe but only if something eventually notices. Relying on the stream alone means the platform's correctness depends on a delivery guarantee it does not control, and the gap is invisible precisely because nothing failed.

Decision

Evaluation has three independent triggers and the event stream is the fastest, not the authoritative one. A reconciliation sweeper compares the catalogue's table version history against the set of evaluated versions on a fixed ten-minute cadence and enqueues anything unjudged. Scheduled evaluation runs regardless, at each tier's cadence. The sweeper's catch rate is a published metric: a rising catch rate means the event path is degrading, which is information the event path itself cannot supply.

How it is realised on Databricks and AWS

The sweeper queries Unity Catalog for versions committed since its last watermark per dataset, left-joins against the verdict store's distinct version ids, and enqueues the difference at normal priority. It is deliberately cheap — a metadata query, not a scan — so it can run every ten minutes across 12,000 datasets. Its own failure is caught by an absence check: no work enqueued by the sweeper for two cycles is itself an alert.

Options weighed

• ChosenEvent stream plus a reconciliation sweep, with the sweep as the guarantee: Keeps event latency and removes the dependency on delivery. Costs a metadata query every ten minutes and a second code path that must produce identical verdicts.
• RejectedEvent stream alone, with strong delivery guarantees: Simpler and depends on a guarantee that holds until it does not. The failure is silent, which is the property this whole architecture is built to avoid.
• RejectedPolling alone: Correct and slow: a ten-minute poll cannot meet a ninety-second Tier 1 verdict target, and polling fast enough to meet it is a load problem at 12,000 datasets.
• RejectedRequire producers to call the service after committing: Moves the guarantee into forty teams' pipeline code, where it will be implemented inconsistently and removed during incidents.

Consequences

What it buys

• A correctness property that does not depend on any broker's delivery guarantee.
• Sweeper catch rate as a metric that makes degradation of the fast path visible — something the fast path cannot report about itself.
• A bounded worst case: ten minutes from an undelivered commit notice to an enqueued evaluation.

What it costs

• A second trigger path that must stay behaviourally identical to the first, or the two produce different verdicts.
• A metadata query every ten minutes across 12,000 datasets, cheap but not free.
• A ten-minute window in which a missed version sits unjudged, tolerable only because 'unknown' already withholds promotion.

Choose differently when

Nothing about a better message broker would flip this; the decision is about not depending on any broker's guarantee. What would change is the cadence, which is a cost-versus-latency dial rather than an architectural choice.

Why it holds up over time

Reconciliation against the source of truth is the pattern that has outlived every generation of messaging, and for the same reason each time: a system that cannot detect its own missed work will eventually have missed work it cannot detect. The sweeper is cheap insurance against a class of bug nobody can enumerate in advance.

Is an assertion arbitrary code the platform runs and hopes finishes, or a declaration the platform must be able to cost and combine before accepting it?

Context

The expressive answer is to let authors write code: a Python function, a Spark job, whatever catches the defect. It maximises what can be expressed and it makes the platform's cost and latency unpredictable per assertion, which at 40,000 assertions means unpredictable in aggregate. It also makes fusion impossible — forty assertions on one table become forty scans, and the cost model collapses. The subtler problem is diagnosis: when an arbitrary function fails, the platform cannot tell whether the data is bad or the function is, and the resulting verdict is unattributable. Meanwhile the vast majority of real assertions — schema shape, row predicates, aggregate bounds, null ratios, referential integrity, distribution comparison — are expressible declaratively and fuse into a single pass.

Decision

Assertions are authored in a declarative DSL and compiled to engine-native query plans. The compiler rejects any assertion it cannot bound: no correlated subquery over an unpartitioned table, no cross join without a declared key, no user-defined function whose cost cannot be estimated. All assertions targeting the same dataset and partition are fused into a single scan wherever their plans permit it, and the achieved fusion ratio is reported per dataset. Metadata-only assertions — schema, freshness — are evaluated without reading data files, and a schema failure short-circuits the remaining assertions for that dataset, because row predicates against a changed schema produce meaningless verdicts.

How it is realised on Databricks and AWS

The DSL compiles to Spark SQL executed on a Databricks SQL warehouse. Compilation produces a plan and an estimated scan volume; the estimate is posted to the pull request (ADR-08) and the plan is cached. Fusion is a rewrite that turns N predicates into N aggregate expressions over one pass, with row-level violations collected as identifiers rather than as rows. Escape-hatch assertions that genuinely need custom logic are supported as a declared, separately budgeted assertion type, priced at full-scan cost and refused on Tier 1 unless explicitly approved.

Options weighed

• ChosenDeclarative DSL compiled to bounded, fusable plans: Makes cost predictable, fusion possible and failures attributable. Costs expressiveness — a genuinely novel check needs the escape hatch or a change to the DSL.
• RejectedArbitrary code per assertion: Maximum expressiveness, no cost model, no fusion, and verdicts that cannot distinguish bad data from a bad check. Right for a small platform with ten assertions and one team.
• RejectedDeclarative DSL with unrestricted SQL passthrough: The compromise that becomes the default: within a year most assertions are passthrough, and the platform has arbitrary code with extra steps.
• RejectedAdopt an existing expectations framework as the authoring surface: Sensible and considered. Rejected because the fusion rewrite and the cost estimator are the two things this design depends on, and neither is available when assertions are opaque function calls — but the DSL's vocabulary is deliberately close, so authors are not learning an alien model.

Consequences

What it buys

• A cost model that still holds at 40,000 assertions, because cost grows with data scanned rather than with rule count.
• One scan per partition instead of forty, with the achieved fusion ratio reported so the saving is visible.
• Verdicts attributable to the data rather than to check code, because the platform knows what each assertion asked.

What it costs

• A DSL to design, document and evolve — a real and permanent maintenance burden.
• A class of author who will find the restriction arbitrary until someone else's unbounded assertion would have consumed their budget.
• An escape hatch for genuinely novel checks, which has to be priced at full-scan cost and watched as a share of the whole.

Choose differently when

If the engine gained genuinely reliable cost estimation for arbitrary functions, the restriction would be worth loosening. The signal to watch is the escape-hatch share: if more than a small minority of assertions need it, the DSL's vocabulary is wrong and the fix is to extend the DSL rather than to widen the hatch.

Why it holds up over time

The principle — declare what you want checked, let the platform decide how — long outlives any particular DSL, and it is the same reason SQL outlived the query languages that let you specify the join order. What will date is the vocabulary, which is why assertions are versioned objects that can be recompiled.

Does evaluation run in the same engine the pipelines use, or in compute dedicated to quality work?

Context

Pushing down into the platform's own engine is cheap, avoids moving 2.4 PB anywhere, and inherits the governance context for free — the evaluator sees exactly what the catalogue's grants allow, with no second access model to maintain. Its defect is timing: quality work then competes for capacity with production pipelines, and the competition is worst during an incident, which is precisely when verdicts matter most. Dedicated compute removes the contention and adds a duplicated cost line, its own credential path into every dataset in the estate, and — if data has to move to reach it — a second copy of regulated data with its own classification problem. The choice turns on whether evaluation is elastic-priority work or carries a hard SLA, and the honest answer is that it is both, at different tiers.

Decision

Evaluation is pushed down into the lakehouse engine; no dataset is copied out of its storage layer to be judged. Contention is managed by lanes rather than by isolation: a reserved-capacity live lane for Tier 1 and Tier 2 work, a shared lane for Tier 3, and a separate low-priority lane on interruptible capacity for retroactive backfill. Per-domain concurrency quotas stop a single domain's rule set from starving the rest, and work is partitioned per dataset so one very large table cannot monopolise the lane.

How it is realised on Databricks and AWS

Databricks SQL warehouses run the live lane with reserved capacity and autoscaling; job clusters handle distribution and reconciliation work; the backfill lane runs on spot instances, where interruption is genuinely acceptable because a delayed retroactive verdict has no consumer waiting on it. Unity Catalog grants are enforced during execution rather than re-implemented, and the evaluator's workload identity is scoped to the assertion's registered targets (ADR-14).

Options weighed

• ChosenPush down, with reserved lanes by tier: No data movement, no second access model, and the contention risk bounded by reservation rather than by isolation. Costs capacity that sits idle between peaks.
• RejectedDedicated evaluation cluster: Clean isolation, duplicated cost, and a standing credential path into every dataset in the estate — which converts a contention problem into a security surface. Right where evaluation carries a hard regulatory SLA that shared capacity cannot hold.
• RejectedPush down with no reservation, best effort: The cheapest option and the one that fails at the worst moment: Tier 1 verdicts arrive late exactly when the platform is unhealthy and the warehouses are saturated.
• RejectedCopy samples out to a validation service: Removes contention by removing completeness. A sampled pass is a different claim from a full-scan pass, and a second governed copy of regulated data is a cost nobody counts at design time.

Consequences

What it buys

• No dataset movement at 2.4 PB, and catalogue grants inherited rather than re-implemented.
• A single access model, so there is no second set of credentials into the estate to secure and rotate.
• Contention bounded by reservation per tier, with backfill on capacity where interruption is genuinely acceptable.

What it costs

• Reserved capacity that sits idle between peaks and shows up as a cost line nobody likes.
• A residual risk left in the open: if the Tier 1 lane cannot be held under incident load, verdict latency degrades exactly when it matters.
• That risk is only measurable during a real incident, not in steady state.

Choose differently when

Two things would flip it: a sustained inability to hold the Tier 1 lane under incident load, or a regulatory requirement that quality evaluation be demonstrably independent of the pipelines it judges. Either justifies dedicated compute and the credential-surface cost that comes with it.

Why it holds up over time

'Computation goes to the data' has been the right default for as long as data has been too big to move, and nothing on the horizon changes the arithmetic. What will change is the lane mechanism — reservations, queues, warehouse pools are all implementation — which is why the decision is stated as lanes by tier rather than as a product feature.

What stops quality coverage from becoming the line item that gets cut, and what happens when a dataset's assertions cannot finish inside their budget?

Context

Data quality spend has a predictable arc: coverage grows, cost grows with it, someone notices the bill, and coverage is cut by whoever can cut it fastest — usually by disabling the expensive assertions, which are disproportionately the full-scan reconciliation and distribution checks that catch silent corruption. The cut is invisible because nothing fails. Meanwhile the per-run failure mode is just as damaging: an assertion set that cannot finish inside its window either delays every other dataset behind it or produces nothing, and 'produces nothing' has already been ruled out as a pass by ADR-05.

Decision

Three controls, at three timescales. At authoring time, an assertion's cost is estimated from its compiled plan and posted to the pull request, so an expensive rule is a conversation before it is an invoice. At run time, each dataset has a declared evaluation budget; exceeding it degrades to the metadata-only assertion set — schema and freshness first — and emits a `degraded` verdict rather than no verdict. In aggregate, total evaluation cost is held at or below 2.5% of the data platform's compute spend, attributed per domain and published alongside each domain's quality score, so that coverage and its price are always read together.

How it is realised on Databricks and AWS

The plan compiler produces an estimated scan volume, converted to an estimate in currency using current warehouse rates and posted as a pull-request comment. Run-time budgets are enforced by the scheduler, which submits the metadata set first and the scan set second, so degradation is a truncation rather than an abort. Cost is attributed by tagging warehouse usage per dataset and rolling it up per domain; the full-scan fraction per dataset is reported so that the cheap path is visibly the default.

Options weighed

• ChosenEstimate at authoring, budget per dataset, cap in aggregate: Puts the price in front of the reviewer, makes exhaustion a graded degradation, and makes the 2.5% ceiling a reviewable event rather than a surprise.
• RejectedNo explicit cost control; optimise when it hurts: Ends in the arc described above: coverage cut under time pressure, starting with the assertions that matter most.
• RejectedHard per-domain chargeback: Aligns incentives sharply and makes quality coverage a budget negotiation per team, which reliably produces least-coverage-that-passes-audit. Worth revisiting only if central capping fails.
• RejectedAbort the whole assertion set on budget exhaustion: Simple, and it produces exactly the absence of information ADR-05 exists to prevent. Truncating to the metadata set keeps the cheapest and most diagnostic checks.

Consequences

What it buys

• Predictable spend, and a price for coverage that is visible at the moment someone reviews the rule.
• Budget exhaustion becomes a graded degradation that keeps schema and freshness, rather than an absence of information.
• Cost published next to the quality score, so coverage and its price are always read together.

What it costs

• Authoring friction, and a cost model that must be maintained as warehouse pricing changes.
• The honest possibility that a dataset's full assertion set does not fit its budget, which surfaces as a coverage decision somebody has to make.
• A 2.5% ceiling that becomes unachievable with full coverage if the fusion ratio proves lower than assumed.

Choose differently when

If the fusion ratio proves low in practice — the prototype's second test — the 2.5% ceiling stops being achievable with full coverage, and the choice becomes explicit: raise the ceiling or reduce Tier 1 coverage. That is a decision for the platform's owners, and the numbers here exist so it is made with numbers rather than with a feeling.

Why it holds up over time

Pricing a control alongside the risk it mitigates is the only way a control survives a cost review, and that will be true in any decade. The specific ceiling is an assumption; the practice of publishing cost next to coverage is the durable part.

When a new or revised assertion is published, does the platform evaluate it against history — and if it fails eighteen of the last thirty versions, who decides whether that is a discovery or a bad rule?

Context

ADR-01 makes retroactive evaluation possible; this decision is about whether it is automatic and what happens to the result. Not backfilling means every rule starts from today and the corruption that has been running for a month stays invisible until someone thinks to ask. Backfilling automatically and then enforcing on the result means a mistyped rule can retroactively condemn a month of a Tier 1 table and mark half its downstream, which is a self-inflicted incident. The two cases are genuinely indistinguishable from the platform's side: a rule that fails eighteen of thirty past versions is either the most valuable thing the platform has produced this quarter or a bug in a predicate, and only someone who knows the domain can say which.

Decision

Publishing an assertion automatically backfills it across a declared lookback window — thirty days for Tiers 1 and 2 — and produces a would-have-failed report. The assertion cannot be promoted from `advise` to an enforcing action until the report has been reviewed and the discovery-or-bad-rule question answered explicitly by the dataset owner. Backfill runs in a separate low-priority lane on interruptible capacity, so re-judging history never delays judging the present. Retroactive verdicts are stored alongside live ones and are clearly marked as retroactive.

How it is realised on Databricks and AWS

Backfill enumerates the dataset's versions inside the lookback window, reads each through Delta time travel, and runs the identical evaluator used on the live path — which is why the four-week prototype is asked to prove the two produce byte-identical verdicts. Results are written as verdicts with `retroactive = true` and summarised into a report posted back to the pull request that introduced the rule. The lane runs on spot warehouses; an interrupted backfill resumes from its watermark.

Options weighed

• ChosenAutomatic backfill, human gate before enforcement: Gets the discovery without the self-inflicted incident. Costs a human in the loop on every new enforcing assertion, which is friction authors will feel.
• RejectedBackfill and enforce automatically on the result: Turns a typo into a month-long retroactive block and a wave of suspect marks. The platform cannot tell a discovery from a bug, so it must not act as though it can.
• RejectedNo backfill; rules apply from publication onwards: The common default, and the reason the silent-corruption case survives: the rule written in response to an incident says nothing about the incident.
• RejectedBackfill on request only: Better than nothing and reliant on someone thinking to ask, at the moment they are least likely to — when they have just written a rule they believe in.

Consequences

What it buys

• The platform's most valuable single capability: a rule written today tells you which of last month's versions to disbelieve.
• A rule's first real feedback is about history, which measurably changes how authors write them.
• Backfill on its own interruptible lane, so the capability costs nothing in live verdict latency.

What it costs

• Compute on every publication, bounded by the lane and the lookback but not zero.
• A mandatory review step before any new assertion may enforce, which authors will feel as a delay.
• A verdict store that grows with backfill volume as well as live volume.

Choose differently when

If the human gate becomes a bottleneck — rules queuing for review for weeks — the right response is a narrower gate (only where the backfill failure rate exceeds a threshold) rather than removing it. If the lookback proves too short to catch the defects that matter, it is a cost dial, not an architectural change.

Why it holds up over time

The capability is durable because it follows from ADR-01 rather than from any tool. The gate is durable for a different reason: no amount of model-assisted rule authoring removes the need for someone who knows the domain to say whether a month of failures is news. If anything, generated assertions make the gate more necessary.

When a corruption window is found in one dataset, how far does the resulting suspicion travel through the lineage graph?

Context

Correctness argues for full transitivity: if a fact table was wrong for three weeks, everything derived from it during those weeks is also suspect, and everything derived from those, and so on. Applied to a real warehouse, one upstream defect in a payments fact table marks several hundred downstream objects, at which point the mark conveys nothing and consumers learn to ignore it — alert fatigue, one layer down, with the same ending. Stopping at one hop is arbitrary and bounded, and it leaves genuinely affected objects two hops away unmarked. Neither option is right; what makes the choice tractable is that the platform can compute the full transitive set as a report without applying it as state.

Decision

State propagation stops at one hop: datasets directly derived from the affected versions are marked `suspect`. The full transitive closure is computed and published as part of the corruption-window report, together with the reports and feature-store snapshots that read the affected versions, and the lineage coverage percentage behind the calculation. Marks are never cleared automatically by a subsequent successful run; they are cleared by re-derivation or by an explicit human dismissal, and an uncleared mark ages as a debt item.

How it is realised on Databricks and AWS

The blast-radius resolver walks Unity Catalog lineage from the affected dataset versions. One-hop results are written to the state index as `suspect` marks with a hop count and a null `cleared_ts`; the transitive set and the consumer list go into the report only. Median mark age is a published metric, because it is the signal that says whether the propagation depth is right.

Options weighed

• ChosenOne-hop state, full transitive report: Keeps the mark meaningful and the information complete. Costs a real gap: an object two hops out is in the report and not in the state, so a consumer reading state alone can miss it.
• RejectedFull transitive propagation as state: Correct and self-defeating at warehouse scale. Worth revisiting once column-level lineage is available, because precision is what makes transitivity survivable.
• RejectedNo propagation; report only: Puts the entire burden on someone reading a report during an incident, and leaves the orchestrator with nothing to act on.
• RejectedPropagate by tier — transitively into Tier 1, one hop elsewhere: Genuinely considered, and the closest alternative. Rejected for the MVP because it makes the mark's meaning depend on the reader's tier, which is hard to explain and therefore hard to trust.

Consequences

What it buys

• A suspect mark that still means something after a year, because it has not been applied to half the warehouse.
• A report that is honest about its own completeness, since lineage coverage is stated alongside the blast radius.
• Median mark age as the signal that says whether the chosen depth is right.

What it costs

• A real two-hop gap: an object two steps downstream is in the report and not in the state, so a consumer reading state alone can miss it.
• Clearing marks is a human workflow, which is the part most likely to be neglected.
• The gap has to be stated on the report rather than hidden, which makes the deliverable less satisfying than it looks.

Choose differently when

Column-level lineage is the flip condition. With it, transitive propagation becomes precise enough to be useful — 'this column in this downstream table is suspect' rather than 'this table is suspect' — and the reason for stopping at one hop disappears. Median mark age climbing is the signal that the current depth is already wrong.

Why it holds up over time

The underlying claim is durable: a signal that fires on everything conveys nothing. What will change is the depth the platform can afford, and it will change as lineage precision improves rather than as graph traversal gets faster.

What protects the platform from its own assertions — a rule that crashes the engine, a rule that fires every night on a legitimate business change, a rule whose author has moved teams?

Context

The failure mode that kills a data quality programme is not missed defects; it is false positives. A rule that fires wrongly twice a week teaches everyone to route its alerts to a folder, and within a month the gate it feeds is advisory in practice while remaining mandatory on paper. Distributional assertions are the worst offenders, because a genuine business change — a new merchant category, a pricing experiment, a festival — looks exactly like a defect to a baseline trained on the previous month. There is also a cruder failure: an assertion whose plan reliably exceeds its budget or crashes the engine, which without isolation takes down the evaluation of every other assertion on that dataset.

Decision

Three structural protections. Per-assertion circuit breaking: an assertion that fails to execute for a threshold of consecutive runs is broken out individually, moved to probation, and its owner notified, while its siblings continue to be evaluated. Probation for every new assertion: thirty days as `advise` only, regardless of its declared action. Automatic demotion: an assertion whose false-positive rate exceeds 5% over a trailing thirty days is demoted to `advise` without waiting for a human, and raised as a debt item. Distributional assertions never leave probation automatically — promotion requires an explicit act after the rate has been measured.

How it is realised on Databricks and AWS

False positives are counted from acknowledged outcomes: a failing verdict that is overridden, dismissed, or followed by no upstream change within a window counts against the rate. The scheduler tracks consecutive execution failures per assertion version and trips the breaker in the work-item layer, so a poison assertion is skipped rather than retried into the ground. Demotion changes the assertion's effective action, never its declared one, so the author's intent stays visible next to the platform's judgement.

Options weighed

• ChosenProbation, per-assertion circuit breaking, automatic demotion on measured FP rate: Protects credibility without depending on anyone's attention. Costs a false-positive definition that is inherently imperfect — inferring intent from what people did after an alert.
• RejectedManual review of noisy assertions: Depends on the attention of the people the noise has already exhausted. Every platform that chooses this ends with a large set of assertions nobody has looked at in a year.
• RejectedNo probation; enforce from publication: Faster to value and one typo away from a retroactive outage. It also removes the window in which the false-positive rate could have been measured at all.
• RejectedSuppress noisy alerts at the notification layer: The tempting shortcut: keep the verdict, stop the page. It produces a platform whose verdicts nobody sees, which is worse than one whose verdicts are demoted honestly.

Consequences

What it buys

• Alerts people still believe, which is the property that decides whether any of the rest of this is enforced.
• A poison assertion is isolated to itself rather than taking down evaluation of every other rule on that dataset.
• Demotion happens without depending on the attention of the people the noise has already exhausted.

What it costs

• A thirty-day delay before a new rule can protect anything.
• A false-positive metric inferred from imperfect signals — what people did after an alert, rather than what they meant.
• The real risk that a correct assertion firing on a genuinely broken upstream is demoted because nobody fixed the upstream, which is why demotion opens a debt item rather than closing the matter.

Choose differently when

If the false-positive inference proves too noisy to act on, the answer is an explicit acknowledgement workflow — someone marks a verdict as a false positive — rather than abandoning automatic demotion. The 5% threshold is an assumption; the mechanism is not.

Why it holds up over time

This is the decision that will matter most when assertions are proposed by models rather than typed by engineers, because the volume of plausible-but-wrong rules will go up by an order of magnitude. A platform that demotes on measured credibility scales to that; one that relies on review does not.

Who may declare that a failing assertion stops a pipeline — and is that declaration made when the rule is written or when it fires?

Context

Two questions that are usually conflated. On authority: an assertion that blocks can stop a Tier 1 dataset, so the power to write one is the power to halt another team's delivery. Granting it to any author is how a platform acquires an outage caused by a well-meaning engineer in a different domain; withholding it from everyone makes the platform advisory. On timing: deciding the action at run time, from the observed violation rate, adapts to reality — a rule could quarantine two rows and block on two thousand. It also means nobody can answer 'what will happen if this rule fails' before it fails, which makes the rule unreviewable, and puts the most consequential decision in the code path least able to be inspected.

Decision

The verdict action — `advise`, `quarantine`, `block` — is a property of the assertion and the dataset tier, declared at authoring time and refused by the tier policy where it is not permitted: no blocking on Tier 3, and no blocking on a distributional assertion alone. Only the dataset's owning team, or a delegate they name, may author a blocking assertion on their dataset, grant an override, or retire an assertion. Any author may propose a rule; the owner's approval is what lets it enforce. The single run-time adaptation is the quarantine ceiling in ADR-15, which is itself declared in advance.

How it is realised on Databricks and AWS

The tier and policy registry answers the authority question; the rule CI pipeline calls it before merge, so a rule declaring `block` on an ineligible dataset never lands. Owner approval is enforced through repository ownership on the rule DSL, and the same registry backs the override API's authorisation check at run time (view 20).

Options weighed

• ChosenAction declared at authoring time; blocking authority with the owner: Makes 'what happens if this fails' a reviewable property and keeps the power to halt a pipeline with the team accountable for it. Costs adaptability and some author frustration.
• RejectedAction decided at run time from the violation rate: Adapts well and cannot be reviewed. The consequential decision ends up in the least inspectable place, and nobody can pre-approve the possibility of a stop.
• RejectedAny author may write a blocking assertion: Maximises coverage growth and eventually produces an outage caused by someone with no context on the dataset they blocked.
• RejectedCentral data governance team owns all blocking rules: Consistent, defensible, and a bottleneck that guarantees thin coverage. It also puts the halt decision furthest from the people who understand the data.

Consequences

What it buys

• Every rule's consequence is stated before it can fire, so 'what happens if this fails' is a reviewable property.
• The team that can be paged is the team that authorised the block.
• A tier policy that refuses ineligible rules at merge rather than at 02:00.

What it costs

• Real friction for a cross-domain author who spots a defect and must get the owner's approval.
• Calibration of the tier policy matters: a high refusal rate is a platform signal, not an author problem.
• Adaptability is given up — a rule cannot decide at run time that this failure is worse than usual.

Choose differently when

If owner approval becomes a bottleneck that visibly suppresses coverage, the change worth making is a standing delegation for specific assertion types — schema and freshness, say — rather than opening blocking authority generally.

Why it holds up over time

Tying the authority to stop a pipeline to the accountability for that pipeline is durable because it is an organisational alignment, not a technical one. What will date is the mechanism — repository ownership today, something else later.

How does an override, a suppression or a demoted rule end — and who is accountable for it while it lasts?

Context

Quality debt does not accumulate because people are careless. It accumulates because an exception granted under pressure has no natural end: the block is overridden at 02:00 to get the report out, the incident closes, the override stays. Multiply by forty domains and two years and the platform has a set of unguarded datasets nobody can enumerate, each individually justified at the time. The same is true of suppressions granted during a migration and of assertions demoted for crying wolf. Represented as configuration flags, all three are invisible after the week they were created; represented as objects with owners and expiries, they become a backlog — which is uncomfortable, and correct.

Decision

Overrides, suppressions and probations are first-class objects with an owner, a justification, and a mandatory expiry; unbounded suppression is not supported by the API. Granting one creates a debt register entry in the same transaction, so there is no way to have the exception without the record. At expiry the state reverts automatically and the debt item ages rather than closing. Aged debt — an override past expiry, a suppression older than thirty days, a quarantine batch past retention, a demoted assertion never reworked, a Tier 1 dataset with no assertions — escalates to the owning team and appears on the domain scorecard.

How it is realised on Databricks and AWS

Overrides and debt items live in the same Aurora database and are written together; the override is recorded in the immutable audit log (ADR-14) before the state index changes. Expiry is enforced by the override service rather than by a consumer checking a timestamp, so the reversion happens whether or not anyone is looking. Scorecards are published per domain and per owner, because an unowned score is never paid down.

Options weighed

• ChosenExpiring objects with mandatory justification, paired with a debt item: Makes every exception end and makes the current set enumerable. Costs friction at the worst moment — during an incident, someone has to type a reason and pick an expiry.
• RejectedOverrides as configuration flags: Zero friction and the origin of every unguarded dataset nobody knows about. The failure is that it works perfectly for a week.
• RejectedExpiry optional, with reporting on long-lived overrides: Reporting on a problem is not ending it. The overrides that most need to expire are exactly the ones whose owners will decline to set an expiry.
• RejectedCentral approval for every override: Stronger and unworkable at 02:00. It also moves accountability away from the team that understands the risk, which is the opposite of what ADR-13 establishes.

Consequences

What it buys

• An enumerable set of current exceptions, which is something most platforms cannot produce at all.
• Automatic reversion at expiry, so an exception ends whether or not anyone is watching.
• A debt backlog that can be managed, aged and assigned rather than discovered during an audit.

What it costs

• Friction at the worst possible moment: during an incident, someone has to type a justification and choose an expiry.
• A scorecard that will look bad for the first few quarters, and will be read as a criticism of teams rather than of history.
• A visible debt register invites argument about its weighting — a better argument than the alternative, but an argument.

Choose differently when

Nothing technical flips this. The risk is erosion: a 'permanent exemption' field added for a genuinely awkward dataset, after which the mechanism is decorative. If such a case is unavoidable, it should be a tier change with the owner's sign-off — a statement about the dataset — not an exception with no end.

Why it holds up over time

Of the sixteen decisions, this is the one most likely to be quietly weakened and the hardest to notice if it is, because nothing breaks when an expiry becomes optional. It is also the one that does most to determine whether the platform still means anything in year three.

How does the platform make failures diagnosable without letting a verdict, its samples or its authorship reveal data the reader is not entitled to see?

Context

A useful failure message names the problem: 'the currency column contains INR for 412 rows in the EU partition'. A useful failure message is also a read of the data, delivered to whoever can see the verdict — which is a broader audience than whoever can see the table. The subtler version is authorship: an engineer who cannot read a salary column can write an assertion asserting salary values fall in a range, publish it, and learn from the pass or fail whether anyone earns above a threshold. Neither problem is exotic, and both are invisible in a design that treats the verdict as metadata rather than as a derived read of the data.

Decision

Four controls. Evaluation runs under a workload identity scoped to the assertion's registered targets, never a standing credential broader than their union. Catalogue access controls are enforced during evaluation, so an author who cannot read a column cannot publish an assertion over it. Sample violating values are redacted by default and emitted only for columns explicitly classified non-sensitive; everything else emits row identifiers rather than contents. And the override audit log — actor, justification, expiry, approver — is written to immutable storage before the state change it authorises, so the record cannot be the thing that failed.

How it is realised on Databricks and AWS

Unity Catalog grants are evaluated at assertion publication and again at execution, and the evaluator's identity is issued per assertion target set rather than held as a service-wide credential. The sample redactor sits in the evaluation plane, before any verdict crosses into the verdict store (view 19). The audit log is S3 Object Lock in compliance mode with a seven-year retention, which the platform's own operators cannot rewrite.

Options weighed

• ChosenScoped identity, grants enforced at authoring and execution, redaction by default, immutable audit: Closes the read channel, the authorship channel and the tampering channel. Costs diagnosability: a failure often says 'these 412 row ids' rather than what was wrong with them.
• RejectedEmit samples freely; restrict who can read verdicts: Moves the problem to access control on verdicts, which then cannot be shared with the people who need to act on them. It also means one misconfigured grant exposes data rather than exposing an outcome.
• RejectedA single service credential with read access to the estate: By far the simplest to operate, and it makes the quality service the most valuable credential in the platform. Its compromise is a full-estate read.
• RejectedAppend-only audit log in the operational database: Adequate until the question is whether an operator altered it. The override log is specifically the artefact an auditor asks for, so it has to be outside the platform's own write authority.

Consequences

What it buys

• A verdict that can be shared widely without carrying the data it judged.
• An override trail that survives an adversarial question, because the platform's own operators cannot rewrite it.
• No standing full-estate credential, so compromising the quality service does not compromise the warehouse.

What it costs

• Harder debugging: engineers will ask for the failing values and the answer will often be row identifiers instead.
• A dependency on column classification being right, since redaction-by-default fails open for a column wrongly marked non-sensitive.
• Per-target identity issuance is more machinery than one service credential, and has to be operated.

Choose differently when

If classification coverage proves unreliable, the response is to narrow the exception further — sample values only for columns explicitly allow-listed by the data owner, rather than for anything not marked sensitive. That is a stricter default, not a different architecture.

Why it holds up over time

Treating derived signals as reads of the underlying data is a discipline that generalises to every observability surface — logs, metrics labels, error messages, model evaluations. The specific redaction rule will change; the recognition that a verdict is a read will not.

When is diverting bad rows the right answer, and at what point does diverting them become a way of hiding a broken upstream?

Context

Quarantine is the most attractive verdict action and the most easily abused. Where a violation is attributable to identifiable rows, diverting them and promoting the rest is genuinely better than blocking: the dataset stays usable, the bad rows are preserved with their context, and the fix is replayable. Two things go wrong. The first is category error — an aggregate failure, a row count out of bounds, a reconciliation gap against the ledger, is a property of the dataset, so there are no offending rows to divert and any attempt to quarantine 'the problem' is arbitrary. The second is scale: diverting two rows in a million is a correction, and diverting a quarter of a batch is a broken upstream being presented as partial success, with the remainder promoted and consumed as though it were complete.

Decision

Quarantine is available only to row-level and referential assertions, where a violation is attributable to identifiable rows. Aggregate, distributional and reconciliation failures may only advise or block. A per-dataset quarantine ceiling — 2% of rows in the batch by default — escalates the verdict to `block` when exceeded. Quarantined rows carry the full verdict context so a fix can be replayed, they inherit the source dataset's classification and masking policy, and re-admission happens only through the normal ingestion path. Rows still unfixed at the end of the retention window expire and are counted as quality debt.

How it is realised on Databricks and AWS

The diverter writes to a Delta quarantine table per source dataset, partitioned by source dataset and date, with assertion id and version, source version, partition and timestamp as columns. Classification and masking are inherited from the source in Unity Catalog rather than configured separately, so quarantine cannot become a lower-governance copy. Expired batches are reported by volume into the debt register (ADR-16) rather than being silently dropped.

Options weighed

• ChosenRow-attributable only, 2% ceiling escalating to block: Keeps datasets usable in the common case and refuses to let quarantine disguise a systemic break. Costs a threshold that is a judgement call per dataset.
• RejectedQuarantine anything that fails: Requires inventing offending rows for dataset-level failures, and produces a promoted remainder that no aggregate assertion would accept.
• RejectedNo ceiling; quarantine whatever fails the predicate: The version that looks fine in every test and hides the real incident: a producer sends a quarter of a batch in the wrong units and the platform quietly promotes the other three quarters.
• RejectedQuarantine into a separate, simpler governance zone: Operationally easier and a compliance defect: an automated process creating a copy of regulated data in a table no domain team thinks of as theirs is exactly how a data protection finding happens.

Consequences

What it buys

• Datasets stay usable in the common case, where a handful of rows are wrong and the rest are fine.
• Quarantine cannot disguise a systemic break as partial success, because the ceiling escalates to block.
• Quarantined data inherits the source's classification, so an automated process cannot downgrade governance.

What it costs

• A ceiling per dataset that will be argued about, and that is a judgement call rather than a derived number.
• A quarantine estate that has to be governed, aged and reported on, in tables no domain team naturally owns.
• Re-admission through ingestion is slower than a direct write — deliberately, and it will still be complained about.

Choose differently when

If a legitimate workload routinely produces more than 2% violating rows — a genuinely noisy third-party feed where the noise is expected — the answer is a dataset-specific ceiling with the owner's sign-off, not the removal of the ceiling. A rising rate of ceiling escalations across a domain is a producer-quality signal, not a threshold problem.

Why it holds up over time

The attribution rule is durable because it follows from the assertion type rather than from policy: an aggregate failure has no rows to divert in any decade. The ceiling is durable as a concept — a correction mechanism must have a point at which it admits the problem is systemic.