Anomaly Detection on Data Pipelines
Why learned thresholds beat static ones for volume and freshness, the base-rate problem that makes naive alerting useless, and how to structure alerts so that people still read them after six months.
Static thresholds on data metrics fail in both directions at once. Set them tight and they fire on every holiday, product launch and marketing campaign. Set them loose and they miss the 30 percent drop that mattered. The response is to learn what normal looks like, which works and introduces its own well-defined problems.
What to model
The metrics worth monitoring automatically are few and consistent across pipelines: row count per partition, freshness or arrival delay, null rate per column, distinct count per key column, and summary statistics for numeric columns. Together these catch most structural failures.
Modelling means capturing the seasonality these metrics genuinely have. Daily volume has weekly seasonality, often monthly, and holiday effects. A seasonal decomposition, or simply comparing to the same weekday over recent weeks, removes most of the false positives that a static threshold generates. Robust statistics matter here: a median and median absolute deviation over recent history is far less sensitive to a single past incident than a mean and standard deviation, which get inflated by the very anomalies you want to detect.
Freshness is better modelled as a survival question than a threshold. Rather than alerting when data is more than two hours late, alert when it is later than it has been on 99 percent of previous days for this day of week, which adapts to a pipeline whose runtime naturally grows.
The base rate problem
This is the reason most anomaly detection on data pipelines gets switched off.
A platform with 1,000 tables running 5 checks each performs 5,000 tests per day. At a 1 percent false positive rate that is 50 alerts daily. If genuine incidents occur twice a week, the precision of the alert channel is under 1 percent, and no human sustains attention on a channel that is wrong 99 times out of 100.
Three responses actually help. Raise the threshold so the false positive rate is far below the incident rate, accepting reduced sensitivity. Aggregate correlated alerts, since one upstream failure firing 40 checks should be one alert naming the root table rather than 40. And route by criticality, so only tables with downstream production consumers page anyone while the rest write to a dashboard.
When it breaks
Learned baselines learn the incident. A model trained on recent history absorbs an ongoing problem as the new normal within days, and the alert stops firing while the problem persists. Excluding periods flagged as incidents from the training window is necessary and requires incidents to be recorded, which most platforms do not do systematically.
Anomaly is not the same as problem. A genuine 40 percent traffic increase from a successful launch is anomalous and correct. Detection cannot distinguish them, so alerts must carry enough context, which other metrics moved, what upstream looks like, for a human to make that call in seconds rather than minutes.
Cold start. New tables have no history, so they are unmonitored exactly when they are least reliable. Bootstrapping from similar tables or from declared expectations covers the gap, and defaulting to no monitoring for the first month is the common and wrong choice.
Monitoring the wrong grain. Total row count across all partitions can look normal while one partition is empty. Checks belong at the grain at which failures occur, which is usually per partition and per source, and aggregating up to a single number is how partial failures hide.
Detection without ownership does nothing. An alert with no named owner and no runbook is noise regardless of how well the model works. The statistical problem is the easy half; the routing and response are what determine whether the system has any effect.
14 flashcards for this concept
Click a card to reveal the answer.