Expectations, Assertions and Where to Put Them
The categories of check a data pipeline can make, why the placement of a check matters more than the check itself, and the failure mode of testing only what is easy to test.
A data quality check is an assertion about a dataset that can fail. Writing them is easy; the difficulty is choosing which assertions are worth having, where in the pipeline they run, and what should happen when one fails. Get those wrong and you have a system that alerts constantly on things nobody acts on while passing the failure that actually mattered.
Categories of check
Schema. Columns exist, types match, nullability holds. Cheap, fast, and catches the class of failure that would otherwise break everything downstream in a confusing way.
Volume. Row counts within an expected range, absolute or relative to recent history. This is the single highest-value check in most pipelines, because a partial load is far more common than a corrupted one and is invisible in every per-row check.
Distribution. Nulls under a threshold, categorical values inside a known set, numeric distributions consistent with recent history. This is where genuine data drift shows up, and where thresholds are hardest to set.
Referential. Foreign keys resolve, joins do not multiply rows, uniqueness holds on keys. Join fan-out from a duplicated dimension row is one of the most damaging silent errors available, because it inflates every aggregate downstream by a plausible-looking amount.
Business rules. Domain invariants: revenue is non-negative, an order's ship date follows its order date, a status transition is legal. These catch errors no generic check can and require domain knowledge to write.
Placement decides value
The same check has completely different value at different points.
At ingestion, checks reject bad data before it enters the platform, so the blast radius is one dataset and the fix is upstream. This is the cheapest place to catch anything and the hardest politically, because rejecting a producer's data means blocking their pipeline.
After transformation, checks catch errors introduced by your own logic rather than by the source. A join that fans out, an aggregation over the wrong grain, a filter that removed too much: none of these are visible at ingestion.
Before publication, checks gate whether consumers see the result. This is where a circuit breaker belongs, because it is the last point at which failing loudly is cheaper than propagating quietly.
The useful principle is that checks should sit at every boundary where responsibility changes hands, since that is where assumptions differ and where a failure becomes someone else's problem.
When it breaks
Alert fatigue kills the system. A check that fires weekly and is acknowledged without action is worse than no check, because it trains everyone to ignore the channel. Every alert needs a defined action, and any check whose failures are routinely acknowledged should be deleted or downgraded to a metric.
Warn-only checks are usually ignored. A quality framework that logs failures without blocking anything produces a dashboard nobody reads. The decision of which checks block and which warn is the important one, and the safe default is that checks protecting downstream consumers block.
Thresholds set from one week of data will fire on the first holiday. Volume and distribution checks need seasonality awareness, or a comparison window long enough to include the normal cycle. Static thresholds on seasonal data are the largest single source of false positives.
Testing what is easy is the real failure. Schema and null checks are simple to write, so pipelines accumulate hundreds of them while the business rules that would catch a genuine error go unwritten because they require asking someone what the data means. A pipeline with 200 schema assertions and no domain invariants is well-instrumented and not well-protected.
12 flashcards for this concept
Click a card to reveal the answer.