practice

Pipeline Anomaly Detection

Monitoring row counts, distributions, freshness and schema for unexplained change, because data pipelines fail silently far more often than they error.

An application failure throws an exception. A pipeline failure frequently completes successfully having processed a fraction of the expected data, and the first indication is a business user noticing a number looks low — typically weeks later.

The four signals that catch most of it are cheap. Freshness: did the table update when expected? Volume: is the row count within its normal range for this day of the week? Schema: did columns appear, disappear or change type? Distribution: did a column's null rate, cardinality or mean shift materially?

Volume and distribution need a baseline, and the baseline has to respect seasonality — Monday is not Sunday, and month-end is not mid-month — or the alerting is noise and gets muted, which is the usual failure.

Where this pays for itself is upstream change nobody announced. A source system deploys, an optional field starts arriving empty, and distribution monitoring catches it the same day rather than after a quarter's reporting is built on it. That single class of incident justifies the whole practice.