Pipeline Anomaly Detection
Monitoring row counts, distributions, freshness and schema for unexplained change, because data pipelines fail silently far more often than they error.
An application failure throws an exception. A pipeline failure frequently completes successfully having processed a fraction of the expected data, and the first indication is a business user noticing a number looks low — typically weeks later.
The four signals that catch most of it are cheap. Freshness: did the table update when expected? Volume: is the row count within its normal range for this day of the week? Schema: did columns appear, disappear or change type? Distribution: did a column's null rate, cardinality or mean shift materially?
Volume and distribution need a baseline, and the baseline has to respect seasonality — Monday is not Sunday, and month-end is not mid-month — or the alerting is noise and gets muted, which is the usual failure.
Where this pays for itself is upstream change nobody announced. A source system deploys, an optional field starts arriving empty, and distribution monitoring catches it the same day rather than after a quarter's reporting is built on it. That single class of incident justifies the whole practice.