ML Observability & Drift
Feature and prediction monitoring, delayed labels, drift statistics, and alerting that does not cry wolf.
5concepts
64flashcards
36minutes of reading
- 01 Drift Statistics and What They Miss PSI, KL divergence, KS and MMD compared on what they detect and where they fail, why per-feature tests miss joint shifts, and the multiple-comparison problem that makes wide monitoring noisy.
- 02 Monitoring Without Labels What to watch when ground truth arrives months late or never, why prediction distributions and confidence are the highest-value proxies, and how to estimate performance from unlabelled data.
- 03 Observability for LLM Applications Why the classical monitoring stack does not transfer to systems with free-text output, what a trace over an agent must capture, and the online quality signals that work without ground truth.