ML Observability & Drift
Feature and prediction monitoring, delayed labels, drift statistics, and alerting that does not cry wolf.
11concepts
101flashcards
76minutes of reading
- 01 Delayed labels and deferred evaluation What to do when ground truth for today's predictions arrives in weeks or months, how label lag distorts monitoring windows, and which proxies are safe to alert on.
- 02 Harmful versus benign shift Why a statistically significant change in the input distribution is neither necessary nor sufficient for a drop in model performance, and what to monitor instead.
- 03 Prediction Logging and Traceability What to record at inference so that a question asked three months later has an answer, why the feature vector matters more than the input, and the sampling and privacy tradeoffs.
- 04 The Three Drifts and How They Differ Covariate shift, label shift and concept drift decomposed precisely, why only one of them necessarily degrades a model, and which of them your monitoring can actually see.
- 05 Drift Statistics and What They Miss PSI, KL divergence, KS and MMD compared on what they detect and where they fail, why per-feature tests miss joint shifts, and the multiple-comparison problem that makes wide monitoring noisy.
- 06 Monitoring Without Labels What to watch when ground truth arrives months late or never, why prediction distributions and confidence are the highest-value proxies, and how to estimate performance from unlabelled data.
- 07 Observability for LLM Applications Why the classical monitoring stack does not transfer to systems with free-text output, what a trace over an agent must capture, and the online quality signals that work without ground truth.
- 08 Sequential monitoring and always-valid alarms Why running a fixed-sample test every day inflates false alarms, and how confidence sequences, e-values and conformal test martingales give bounds that stay valid under continuous peeking.
- 09 Shift attribution and incident triage Once a monitor fires, how to decompose the performance change across data quality, covariate shift and concept change, and the order of checks that resolves most model incidents fastest.
- 10 Slice-Based Monitoring and Alert Design Why aggregate model metrics hide failures on important subgroups, how sample size and multiple testing limit what per-slice monitoring can detect, and how to route slice alerts so they stay actionable.
- 11 Unlabelled accuracy estimation How confidence thresholds, confusion-matrix inversion and model disagreement estimate a deployed model's current error rate without any ground truth, and which assumption each one buys that estimate with.