Delayed labels and deferred evaluation

A loan issued today is charged off or repaid over two years. A churn prediction resolves after the contract period. A medical risk score is validated when a diagnosis is coded, which may be months later and only for patients who returned. In all three cases the metric that defines model quality is unavailable for the entire period during which a regression could be doing damage, and the monitoring system has to work in the gap.

Label lag is not just slowness. It systematically distorts what a naive dashboard shows.

The maturation bias

If you compute accuracy over "predictions made in the last 30 days", you are computing it over the subset of predictions whose labels have already matured, and that subset is not random. Fast-resolving cases are different from slow-resolving ones: fraud confirmed within hours is blatant fraud, chargebacks that take 90 days are disputes. A metric on matured labels is biased toward the fast tail, and it improves or degrades for reasons that have nothing to do with the model.

Two fixes, both partial. Cohort by prediction date, not by label date, and only report a cohort once its label maturity exceeds some fixed fraction, accepting that the freshest number you can trust is old. Or model the maturation curve explicitly, treating unmatured outcomes as censored and using survival-style estimators, which gets you a current number with a confidence interval wide enough to be honest.

Proxy signals and their error bars

In the gap, monitoring runs on proxies. The usable ones, roughly in order of trustworthiness:

• Partially observed outcomes. A 7-day early indicator that historically correlates with the 90-day label. You can quantify the correlation on history, which is what makes this the best option.
• Human-labelled audit samples. Expensive, small, unbiased. A few hundred adjudicated cases a week gives a noisy but assumption-free signal that anchors everything else.
• Unlabelled accuracy estimators. Confidence-based or disagreement-based estimates of current risk, which carry assumptions that may be violated by the very shift you are trying to detect.
• Output distribution statistics. Cheap, immediate, weakly related to performance.

The discipline is to publish the proxy's historical relationship to the real metric alongside the proxy. "Approval rate rose 4%" means nothing; "approval rate rose 4%, and in the last eight quarters a move of that size preceded a default-rate change of between -0.2 and +1.1 points" is a monitoring signal.

Feedback loops make the delay worse

Delayed labels are usually also selectively observed labels. You only learn the outcome for applicants you approved, items you showed, alerts an analyst reviewed. The observed label distribution is the model's own policy applied to the world, so a model that has quietly become more conservative will look increasingly accurate on the shrinking set it acts on. Breaking this needs deliberate exploration: a small randomised holdout that bypasses the model, or propensity weighting against logged scores, both of which cost something real and are the only way the monitoring signal stays connected to the population rather than the policy.

When it breaks

The trap is a monitoring system that is silent precisely because the label pipeline broke. An upstream join that drops outcome rows looks identical to a model whose predictions all came true. Alert on label arrival rate and maturity curve shape as first-class signals, not just on the metrics computed from them. A monitoring stack that cannot distinguish "no bad outcomes" from "no outcomes" will sleep through its worst incident.