A dashboard was wrong for three weeks before anyone noticed. What monitoring would have caught it?

What would have caught it

• Freshness assertions per dataset. A pipeline that stopped produces no error and yesterday's data — the most common cause of a quietly wrong dashboard and the cheapest to detect.
• Row-count anomaly detection against the historical pattern, which catches a partial load. Structural tests pass on a partial load and every aggregate is wrong.
• Distribution checks on key columns, which catch an upstream semantic change — same schema, different values, no error.
• Reconciliation against an authoritative source, which is the only detection for a transformation bug producing plausible values.
• Lineage, so when something is detected, the affected downstream assets are identifiable rather than guessed at.

Why it went unnoticed for three weeks

Because nothing failed. Data quality problems are usually silent: the pipeline succeeded, the schema was valid, the numbers were plausible. The absence of an error is not evidence of correctness, and monitoring built around errors cannot see it.

And because nobody owned the dataset. A number that looks slightly wrong with no owner is a number nobody investigates.

What to monitor and what to skip

Concentrate on the datasets that decisions are made on. A build running a thousand assertions across everything spends its time in the wrong place and produces a failure rate that trains the team to ignore it — the same dynamic as flaky tests and noisy alerts.

The response question that must be answered in advance

Should a failing check stop the pipeline or quarantine and continue? Both are defensible: stopping prevents bad data from propagating; continuing keeps the platform available with a visible gap.

The default is usually neither — the check fails, an alert is raised, and the pipeline continues silently, which is the worst of both.

The organisational half

A named owner per dataset with an agreed response. A check that fails, blocks nothing and has no owner is a report — and reports about data quality accumulate unread at exactly the same rate as everything else that nobody owns.