A data platform has thousands of datasets and cannot write manual quality tests for all of them. What should the observability approach be?

The layering that works

Automated baseline everywhere, explicit assertions where correctness matters.

• Automated, on every dataset: freshness, volume against the historical pattern, schema change detection, null-rate and distribution shift. These need no per-dataset configuration and catch the majority of silent failures.
• Explicit assertions, on the datasets that matter: uniqueness, referential integrity, business invariants, cross-system reconciliation. Expensive to write, so spend them where being wrong is costly.

Attempting explicit tests everywhere means most datasets get none, because the effort is never prioritised.

Making alerts actionable

• Lineage-aware grouping. One upstream failure causes anomalies in every downstream dataset; alerting on each produces a storm that obscures the cause. Alert on the root and list the affected.
• Severity from consumption, not from position. A dataset feeding a regulatory report and a dataset feeding an abandoned dashboard should not page equally — and consumption data is what distinguishes them.
• Seasonality in the baseline, or every Monday produces false positives and the alerts get muted.
• A stated response for each alert type. An anomaly alert with no defined action trains people to ignore it, and muted alerts are worse than absent ones because they create false confidence.

The metric to hold it to

Proportion of data incidents detected by monitoring rather than reported by a consumer. That number starting low and rising is the only evidence the investment is working.