Seasonal-Hybrid Decomposition for Anomalies
Why anomaly detection on a seasonal series must remove the seasonality first, how robust decomposition avoids letting anomalies distort the fit, and the residual tests that follow.
Traffic drops 60 percent every night and 30 percent every weekend. An anomaly detector on the raw series fires at 2am daily and all weekend, and misses a genuine 20 percent drop on a Tuesday afternoon because it is well within the range the series covers. Detection has to happen on what remains after the expected pattern is removed.
Decompose, then test the residual
The structure is: model trend and seasonality, subtract them, and apply anomaly detection to the residual. If the decomposition is good, the residual is approximately stationary and roughly symmetric, which is what every statistical test assumes and what the raw series never satisfies.
STL, seasonal-trend decomposition using loess, is the standard tool. It handles multiple seasonal periods, allows the seasonal shape to evolve rather than assuming it is fixed, and is robust to outliers when run in its robust mode. That last property is not incidental: the anomalies you want to detect will otherwise contaminate the seasonal estimate and hide themselves.
The circularity problem
Fitting a decomposition on data containing anomalies produces a trend and seasonal component that partly absorb them, shrinking the residual at exactly the points of interest. This is the central difficulty and there are three standard responses.
Use robust fitting, so extreme points have bounded influence on the estimate. Use the median rather than the mean for the trend, for the same reason. Or iterate: fit, detect, remove the detected points, refit on the cleaned series, and re-detect.
The seasonal-hybrid ESD approach combines robust decomposition with an extreme studentised deviate test on the residual, and its contribution is precisely this combination: median-based decomposition so anomalies do not distort the baseline, followed by a test with a controlled false positive rate for multiple simultaneous outliers.
Testing the residual
Generalised ESD tests for up to \(k\) outliers with a controlled overall error rate, which is the right shape when several anomalies may occur in a window. A plain z-score threshold has no such control and is distorted by the outliers it is testing for, since they inflate the standard deviation.
Robust scaling using the median absolute deviation instead of the standard deviation is the minimum correction if a simple threshold is used.
When it breaks
Multiple seasonalities interact. Daily, weekly and annual patterns plus holidays that move between years exceed what a single decomposition handles cleanly. Explicit holiday regressors, or a model that accepts them, are needed rather than expecting the seasonal component to absorb them.
A level shift is not an anomaly. A permanent change in the baseline produces a long run of large residuals, so an outlier detector reports hundreds of anomalies where a changepoint detector would report one event. Running both, and treating a sustained run of outliers as a changepoint signal, is what distinguishes them.
Short histories cannot support seasonal estimation. Estimating an annual pattern needs several years, and a series with eight months has no basis for one. Borrowing the seasonal shape from related series is the practical response and imports whatever those series' patterns are.
Residual variance is often not constant. Many metrics have variance proportional to level, so a fixed residual threshold is too sensitive at peak and too lax at trough. Log-transforming before decomposition, or scaling the threshold by the fitted level, corrects it.
12 flashcards for this concept
Click a card to reveal the answer.