Slice-Based Monitoring and Alert Design

A fraud model's overall recall holds at 91 percent for a month. Recall on transactions from one newly integrated payment provider, 0.6 percent of volume, has fallen to 40 percent since the integration, and the aggregate cannot move far enough to notice. In medical imaging this pattern has a name, hidden stratification: models with strong overall performance showed relative performance differences of over 20 percent on clinically important subsets that neither training nor testing identified (Oakden-Rayner et al., 2020, Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging, ACM CHIL, arXiv:1909.12475).

Slicing the monitor is the obvious response. The difficulty is that a monitor with hundreds of slices is a statistical instrument with hundreds of false-alarm sources and some slices too small to measure anything. Drift statistics and what they miss treats multiplicity for input features; this concept deals with outcome metrics per slice and with the alerting design that decides whether anyone acts.

What a slice can detect

For a slice \(s\) with \(n_s\) labelled predictions in a window and baseline error rate \(e_s\), the standard error of the observed rate is \(\sqrt{e_s(1-e_s)/n_s}\). The smallest rise you can flag at \(z\) standard errors is roughly

With 400 labelled predictions a day and a 5 percent baseline, the standard error is 1.1 points, and at \(z = 3\) only a rise of about 3.3 points is detectable, a 66 percent relative increase. The same model's aggregate over a million predictions detects changes of a few hundredths of a point. Detection power is spent where volume is, which is the opposite of where hidden failures live.

Three responses follow. Lengthen the window for small slices, accepting slower detection. Pool related slices hierarchically, so a small slice borrows strength from its parent. Or declare explicitly that a slice is unmonitorable at current volume and cover it with periodic labelled audits instead of alerts.

Multiplicity

With \(m\) independent slice tests at level \(\alpha\), the chance that at least one fires under no real change is \(1-(1-\alpha)^m\). At \(m = 500\) and \(\alpha = 0.01\) that is above 99 percent, with an expected five false alerts every evaluation cycle.

Bonferroni, testing each at \(\alpha/m\), controls the probability of any false alert and at \(m = 500\) destroys power for small slices. The Benjamini-Hochberg procedure controls the false discovery rate instead: sort the \(m\) p-values ascending as \(p_{(1)} \le \dots \le p_{(m)}\), find the largest \(k\) with

and flag the \(k\) smallest (Benjamini & Hochberg, 1995, Controlling the False Discovery Rate, JRSS B 57(1):289-300). With \(q = 0.1\), roughly one in ten flagged slices is expected to be noise. That is a reasonable contract for an investigation queue and a poor one for waking someone up. The original guarantee assumes independent or positively dependent tests; nested slices such as country and country-by-device are positively dependent, which is usually tolerable, but the dependence should be a known assumption rather than an accident.

Which slices, and who hears about them

Predeclared slices come from product and risk knowledge: regions, customer tiers, languages, protected groups, input sources, new integrations. They are interpretable and miss what nobody anticipated. Automated slice discovery searches for coherent, large, underperforming subsets; Slice Finder does this for interpretable feature conjunctions (Chung et al., 2018, Automated Data Slicing for Model Validation, arXiv:1807.06068), and Domino clusters cross-modal embeddings, recovering 36 percent of 1,235 synthetic slices in its benchmark (Eyuboglu et al., 2022, Domino, ICLR, arXiv:2203.14960). Practitioners disagree on the balance. Discovery finds unknown failures and turns every run into a multiple-testing problem over an enormous implicit hypothesis space; predeclared slices keep inference honest and stay blind. A workable split uses predeclared slices for alerts and discovery for weekly review.

Routing matters as much as statistics. Google's SRE guidance holds that every page should be actionable and should indicate users are being affected (Ewaschuk, Monitoring Distributed Systems, in Google SRE Book). Translated to slices, page only when a slice with material business or harm impact shows an effect large enough to act on, sustained across consecutive windows. Everything else, including FDR-controlled discoveries, becomes a ticket with the slice definition, sample size and interval attached.

When it breaks

Intersections explode. Ten attributes with five values each give \(\binom{10}{2} \times 25 = 1{,}125\) pairwise slices before any three-way cut, most with too few samples to measure. Monitoring every intersection guarantees noise; monitoring none misses failures that exist only at intersections.

Labels arrive late and selectively. Slice outcome metrics lag by the label delay, and if labels come only from cases the model acted on, small slices are measured on a filtered population. Proxy signals per slice, such as prediction rate and override rate, fill the gap.

Significance without magnitude. Large slices produce tiny p-values for trivial changes. Every alert rule needs an effect-size floor as well as a significance condition.

Suppression hides real incidents. Snoozing a noisy slice alert is rational and removes the monitor. Snoozes should expire, and a slice silenced three times needs a redesigned rule, not a fourth snooze.