Silent Data Failure

The characteristic data failure is not a crash. The pipeline succeeded, the schema was valid, and the numbers were plausible — and the dashboard was wrong for three weeks.

Monitoring built around errors cannot see it, because there was no error. The absence of an error is not evidence of correctness, and in data pipelines that distinction is the whole problem.

Why it matters

The failure is discovered by someone noticing that a number looks slightly wrong, which requires both that someone is looking and that they are familiar enough to notice. Neither is reliable, and the delay means decisions have been made on the wrong number in the meantime.

Implementation patterns

The checks that catch it, roughly by value:

• Freshness assertions per dataset. A pipeline that stopped produces no error and yesterday's data — the most common cause of a quietly wrong dashboard and the cheapest to detect.
• Row-count anomaly detection against the historical pattern, which catches a partial load. Structural tests pass on a partial load and every aggregate is wrong.
• Distribution checks on key columns, which catch an upstream semantic change — same schema, different values, no error.
• Reconciliation against an authoritative source, the only detection for a transformation bug producing plausible values.
• Lineage, so once something is detected the affected downstream assets are identifiable rather than guessed at.

And the organisational half:

• A named owner per dataset, since a slightly-wrong number with no owner is a number nobody investigates.
• An agreed response: should a failing check stop the pipeline or quarantine and continue? Both are defensible — and the default is usually neither: the check fails, an alert is raised, the pipeline continues silently, which is the worst of both.
• Concentration on the datasets decisions are made on, since a thousand assertions across everything produces a failure rate that trains the team to ignore it.

Industry example

Retail and commerce platforms such as Nykaa and Myntra run reporting that drives merchandising, pricing and inventory decisions, where a three-week-wrong number has already changed what was ordered. The detection that pays for itself first is freshness, because the stopped-pipeline case is both the most common and the least ambiguous.

Failure scenarios

• Error-based monitoring only, blind to the entire class.
• No freshness check, so a stalled pipeline serves stale data indefinitely.
• No ownership, so a noticed anomaly is not investigated.
• Checks that fail and block nothing, producing reports.
• Assertions spread evenly across all datasets, producing noise then blindness.

Trade-offs

Anomaly detection produces false positives — a legitimate business change looks like a data problem — and a noisy check trains the team to ignore it exactly as a flaky test does.

The mitigation is tuning against the seasonal baseline rather than a fixed threshold, and routing anomaly alerts to a channel where judgement can be applied rather than to a pager. Freshness and reconciliation are the checks worth alerting hard on, because their false-positive rate is low and their failures are unambiguous.

Interview question

"A dashboard has been wrong for three weeks and every pipeline reported success. Give me the four checks that would have caught it, ranked by how quickly they would have fired."