Data Quality & Contracts
Expectations and assertions, schema evolution, producer-consumer contracts, and detecting silent corruption.
5concepts
62flashcards
35minutes of reading
- 01 Anomaly Detection on Data Pipelines Why learned thresholds beat static ones for volume and freshness, the base-rate problem that makes naive alerting useless, and how to structure alerts so that people still read them after six months.
- 02 Detecting Silent Corruption The failures that produce structurally valid, plausible, wrong data, why per-row checks cannot find them, and the reconciliation techniques that can.
- 03 Schema Evolution and Compatibility Modes The precise definitions of backward, forward and full compatibility, why the direction depends on whether you deploy producers or consumers first, and the changes that are always breaking.