Evaluating Anomaly Detectors

Anomaly detection evaluation has a documented history of protocols that flatter methods, and the most widely used one is now known to make a random detector look excellent. Reading results in this area requires knowing which protocol produced them.

Why per-timestamp metrics mislead

Real anomalies are intervals, not instants: an outage lasts twenty minutes, a degradation lasts an afternoon. Computing precision and recall per timestamp then measures the wrong thing in two ways. A detector that flags one point inside a long anomaly has detected it operationally and scores near zero on recall. A detector that flags a long window overlapping a short anomaly scores well on recall for a mostly wrong alert.

The operational question is whether each anomalous event was detected, with acceptable delay, and how many alerts were raised that corresponded to no event. Those are event-level quantities.

The point-adjustment problem

The common response was point adjustment: if any point within a ground-truth anomaly window is flagged, mark the entire window as correctly detected, then compute per-point metrics.

This is far more generous than it appears. Under point adjustment, a detector emitting random alerts at a modest rate will hit at least one point in most long anomaly windows, and each hit is credited with the whole window. Analyses have shown that random or trivially simple detectors achieve high F1 under this protocol on standard benchmarks, which means published comparisons using it do not establish what they appear to.

The consequence is that a substantial body of reported results is not interpretable at face value, and the protocol has to be checked before any number is compared.

What to report instead

Event-level detection rate: what fraction of ground-truth anomaly events were detected at all.

Detection delay: how long after onset, since a detection after the incident resolved is not a detection in any useful sense.

False alarms per unit time: alert count per day or per week when no anomaly is present, which is the quantity that determines whether the system is usable and is directly comparable to an alert budget.

Range-based precision and recall, which credit partial overlap explicitly with stated weighting for existence, size and position, rather than through an all-or-nothing adjustment.

When it breaks

Labels are unreliable. Anomaly ground truth is usually annotated after the fact by people who may have missed events, and an unlabelled anomaly counts as a false positive against a detector that found it. Benchmark labels in this area have documented quality problems.

Benchmarks are trivially easy. Several widely used anomaly benchmarks contain anomalies detectable by a simple threshold on the raw value, so a sophisticated method beating a naive baseline by a small margin on them establishes very little.

The class imbalance is extreme. With anomalies at a fraction of a percent, accuracy is meaningless and even F1 is dominated by the majority behaviour. Precision-recall curves and the alert budget framing are more informative than any single number.

Evaluation must be online. A method that uses future data to normalise, to fit a threshold, or to select parameters is not evaluating what a deployed detector could do, and this leaks in subtly through preprocessing more often than through the detector itself.