ML Observability & Drift
Feature and prediction monitoring, delayed labels, drift statistics, and alerting that does not cry wolf.
11concepts
101flashcards
76minutes of reading
- 01 Drift Statistics and What They Miss PSI, KL divergence, KS and MMD compared on what they detect and where they fail, why per-feature tests miss joint shifts, and the multiple-comparison problem that makes wide monitoring noisy.
- 02 Monitoring Without Labels What to watch when ground truth arrives months late or never, why prediction distributions and confidence are the highest-value proxies, and how to estimate performance from unlabelled data.
- 03 Observability for LLM Applications Why the classical monitoring stack does not transfer to systems with free-text output, what a trace over an agent must capture, and the online quality signals that work without ground truth.
- 04 Sequential monitoring and always-valid alarms Why running a fixed-sample test every day inflates false alarms, and how confidence sequences, e-values and conformal test martingales give bounds that stay valid under continuous peeking.
- 05 Shift attribution and incident triage Once a monitor fires, how to decompose the performance change across data quality, covariate shift and concept change, and the order of checks that resolves most model incidents fastest.
- 06 Slice-Based Monitoring and Alert Design Why aggregate model metrics hide failures on important subgroups, how sample size and multiple testing limit what per-slice monitoring can detect, and how to route slice alerts so they stay actionable.
- 07 Unlabelled accuracy estimation How confidence thresholds, confusion-matrix inversion and model disagreement estimate a deployed model's current error rate without any ground truth, and which assumption each one buys that estimate with.