ML Observability & Drift
Feature and prediction monitoring, delayed labels, drift statistics, and alerting that does not cry wolf.
5concepts
64flashcards
36minutes of reading
- 01 Prediction Logging and Traceability What to record at inference so that a question asked three months later has an answer, why the feature vector matters more than the input, and the sampling and privacy tradeoffs.
- 02 The Three Drifts and How They Differ Covariate shift, label shift and concept drift decomposed precisely, why only one of them necessarily degrades a model, and which of them your monitoring can actually see.
- 03 Drift Statistics and What They Miss PSI, KL divergence, KS and MMD compared on what they detect and where they fail, why per-feature tests miss joint shifts, and the multiple-comparison problem that makes wide monitoring noisy.
- 04 Monitoring Without Labels What to watch when ground truth arrives months late or never, why prediction distributions and confidence are the highest-value proxies, and how to estimate performance from unlabelled data.
- 05 Observability for LLM Applications Why the classical monitoring stack does not transfer to systems with free-text output, what a trace over an agent must capture, and the online quality signals that work without ground truth.