Data Quality & Contracts
Expectations and assertions, schema evolution, producer-consumer contracts, and detecting silent corruption.
6concepts
69flashcards
43minutes of reading
- 01 Data Contracts as Producer Obligations Why moving the schema definition to the producer changes the economics of data quality, what a contract must contain beyond field types, and the organisational conditions under which contracts actually work.
- 02 Expectations, Assertions and Where to Put Them The categories of check a data pipeline can make, why the placement of a check matters more than the check itself, and the failure mode of testing only what is easy to test.
- 03 Testing Data Transformations How to test the logic of a SQL or dataframe transformation rather than the data flowing through it, using fixture-based unit tests, property-based tests that state invariants, and reconciliation checks that compare what went in with what came out.
- 04 Anomaly Detection on Data Pipelines Why learned thresholds beat static ones for volume and freshness, the base-rate problem that makes naive alerting useless, and how to structure alerts so that people still read them after six months.
- 05 Detecting Silent Corruption The failures that produce structurally valid, plausible, wrong data, why per-row checks cannot find them, and the reconciliation techniques that can.
- 06 Schema Evolution and Compatibility Modes The precise definitions of backward, forward and full compatibility, why the direction depends on whether you deploy producers or consumers first, and the changes that are always breaking.