Data Observability
Freshness, volume, schema and distribution monitoring for pipelines that fail silently.
5 to work through
-
intermediate
A dashboard has shown the same numbers for eleven days. The pipeline reports success every night. What monitoring was missing?
2 min answer -
intermediate
A dashboard was wrong for three weeks before anyone noticed. What monitoring would have caught it?
2 min answer -
intermediate Multiple choice
A grocery marketplace of the kind Instacart operates ingests retailer inventory feeds hourly and deduplicates on store id plus SKU. A retailer changes its SKU format and about 3% of distinct products now collide, so rows silently merge. Ingested row counts fall 3%, inside the 20% anomaly band. Schema tests pass and availability on the storefront quietly degrades. Which single check catches this?
3 min answer -
intermediate
A quarterly board report shows a category down 40%. Investigation finds an upstream system stopped sending a field three months ago. Nothing alerted. What do you change?
2 min answer -
advanced
A data platform has thousands of datasets and cannot write manual quality tests for all of them. What should the observability approach be?
1 min answer
3 terms in this topic
Data Observability
Continuous automated monitoring of data health — freshness, volume, schema, distribution and lineage — designed to detect silent data failures.
practicePipeline Anomaly Detection
Monitoring row counts, distributions, freshness and schema for unexplained change, because data pipelines fail silently far more often than they error.
conceptSilent Data Failure
A data problem that produces no error - the pipeline succeeded, the schema was valid, the numbers were plausible - which is why error-based monitorin…
1 artifact you would hand over
Neighbouring topics
Data Governance & Semantics
General material on ownership, meaning, quality and control of data at enterprise scale.
Data Mesh
Domain ownership, data as a product, self-serve platform, and federated governance.
Data Products
A dataset with an owner, an interface, an SLO, and consumers who can rely on it.
Data Contracts
Producers committing to schema, semantics and freshness, and breaking builds when they do not.
Data Catalog
Discovery, ownership and technical metadata, and why catalogues go stale.
Business Glossary
Agreeing what a term means before arguing about which number is right.
Semantic Layer
Metric definitions held once and served to every tool that asks.
Master Data Management
One authoritative record for a customer or product across systems that each have their own.
Reference Data
Code lists, hierarchies and currencies — small, shared, and quietly load-bearing.
Data Quality Dimensions
Completeness, accuracy, timeliness, consistency, validity and uniqueness as testable claims.
Data Stewardship
The operating model that makes ownership a role rather than a slide.
Data Access Models
Role, attribute and purpose-based access over analytical data, and how they compose.
Row & Column-Level Security
Restricting slices of a table rather than the whole table, and where it is enforced.
Tokenisation & Masking
Dynamic masking, deterministic tokens, and preserving joinability without exposure.
Retention & Purge
Deleting from an append-only estate, and proving the deletion happened.
Data Sharing & Clean Rooms
Collaborating on data neither party may hand over, with computation as the interface.
Sensitivity Labelling
Propagating a classification through joins and derived tables so controls follow the data.
BI Governance
Dashboard sprawl, certified reports, and the number the board is allowed to see.
Self-Service vs Governed
Letting analysts move fast without four teams reporting four different revenues.