ML Observability & Drift
Feature and prediction monitoring, delayed labels, drift statistics, and alerting that does not cry wolf.
11concepts
101flashcards
76minutes of reading
- 01 Delayed labels and deferred evaluation What to do when ground truth for today's predictions arrives in weeks or months, how label lag distorts monitoring windows, and which proxies are safe to alert on.
- 02 Harmful versus benign shift Why a statistically significant change in the input distribution is neither necessary nor sufficient for a drop in model performance, and what to monitor instead.
- 03 Prediction Logging and Traceability What to record at inference so that a question asked three months later has an answer, why the feature vector matters more than the input, and the sampling and privacy tradeoffs.
- 04 The Three Drifts and How They Differ Covariate shift, label shift and concept drift decomposed precisely, why only one of them necessarily degrades a model, and which of them your monitoring can actually see.