A fraud model has been in production for eight months. Ground truth arrives weeks later. How do you know if it is still working?

What the interviewer is testing

Whether you can monitor a model when the outcome metric is delayed, which is the normal case rather than an edge case.

The layered approach

Performance is the lagging confirmation. Input monitoring is the leading indicator, and it is what tells you something is wrong before the labels arrive.

Feature distribution drift. Compare the live distribution of each input feature against the training distribution. A shift means the model is operating on a population it was not validated for. This catches upstream pipeline changes, new customer segments and changed source semantics.

Prediction distribution drift. If the score distribution shifts materially, something changed even if you cannot yet say whether the model is wrong.

Proxy signals available immediately: manual review override rate, customer complaint rate, downstream action rates.

Delayed performance, computed as labels arrive, on a rolling window — the eventual truth.

The distinction that matters

Data drift is a change in the input distribution. Concept drift is a change in the relationship between inputs and outcome — the same features now predict a different result, which is exactly what happens when fraud tactics change. Concept drift is invisible to input monitoring and only appears in performance, which is why the delayed metric cannot be dispensed with.

The control that makes it actionable

A holdout or challenger. Route a small fraction of traffic to a control — a simple rule set, or a prior model version — so you have a comparison baseline rather than an absolute number drifting without context.

What a strong answer adds

The feedback loop problem specific to fraud: the model blocks transactions, so you never learn whether those would have been fraudulent. The training data becomes biased by the model's own decisions. Deliberately allowing a small random sample through, within a risk-controlled value limit, is the standard remedy and it must be designed in — retrofitting it means an argument with risk during an incident.

Common weak answers

Waiting for the labels. Monitoring accuracy alone, which conflates the two drift types.