Detecting Silent Corruption

A pipeline that crashes is a good pipeline. The dangerous failures produce output that is complete, correctly typed, plausibly distributed and wrong, and they survive for months because every check the platform runs is a check the data passes.

What silent corruption looks like

Join fan-out. A dimension table acquires a duplicate key, a join that was one-to-one becomes one-to-many, and every fact row is counted twice. Revenue doubles. Nothing is malformed; there are simply more rows than there should be, and if the duplication rate is modest the number looks like growth.

Silent truncation and coercion. A string exceeding a column's width is truncated. A timestamp with a subsecond component is coerced to second precision. A numeric overflow wraps. Each is a valid value that is not the value that was sent.

Partial loads. A source system's export fails halfway. The load succeeds with 60 percent of the rows, every one of which is correct. Row-level checks pass unanimously.

Timezone drift. A pipeline stage interprets naive timestamps as local time where the previous stage wrote UTC. Every timestamp shifts by a fixed offset, distributions look normal, and daily aggregates are wrong at the boundaries only.

Unit changes upstream. A field moves from cents to dollars, or metres to feet. The distribution shifts by a constant factor and every downstream calculation is off by it.

The common property is that no individual row is invalid. Detection has to be at the aggregate level or by comparison against an independent source.

Techniques that work

Reconciliation against the source. Compare row counts and the sums of key numeric columns against the originating system, not against the previous run of the same pipeline. This is the only check that catches partial loads and unit changes reliably, and it requires the source to expose a control total, which is a request worth making.

Cross-system agreement. The same quantity computed by two independent paths, for example revenue from the transactional database and from the analytics pipeline, should agree within a tolerance. Divergence is the signal, and it catches errors in either path without knowing which.

Cardinality invariants. Assert that a join does not increase the row count, that a primary key is unique, and that the distinct count of an entity column matches expectation. Fan-out is caught by exactly one assertion, before and after the join, and it is one of the highest-value checks available.

Distribution monitoring with history. Track per-column statistics over time and alert on shifts relative to the column's own history rather than to a static threshold. A mean shifting by a factor of 100 is a unit change; a null rate jumping from 2 percent to 40 percent is an upstream failure.

Reconstruction sampling. Periodically take a small sample of output rows and recompute them from raw inputs by an independent path. It is cheap, it catches logic errors nothing else will, and almost nobody does it.

When it breaks

Comparing to yesterday's pipeline output validates nothing. If the corruption is systematic, both runs contain it and agree perfectly. Reconciliation must be against an independent source, and self-consistency checks give false confidence precisely when a persistent bug is present.

Tolerances hide small persistent errors. A one percent tolerance permits a one percent error forever, and a one percent revenue error is material. Tolerances should be set from measured noise, and any check whose tolerance was widened to stop it alerting has been disabled rather than tuned.

Detection lag compounds. Corruption found three months later means three months of downstream artefacts to rebuild, models trained on bad data, and decisions already taken. The value of a check is a strong function of how quickly it fires, which argues for cheap checks on every run over thorough checks weekly.

Some corruption is only visible in the target metric. A subtly wrong feature shows up as a model that performs slightly worse, with no data quality check firing at all. Monitoring model performance is part of the data quality system whether or not the org chart says so.