Metric Ladders from Offline to Online
The chain from a model metric to a business outcome, why each link is weaker than teams assume, and how to validate the links rather than assuming them.
Every AI product runs on an implicit chain: the model metric improved, so the system got better, so users had a better experience, so the business outcome improved. Each arrow is an assumption, each is frequently false, and the chain is rarely written down where it could be checked.
The rungs
Model metrics. Accuracy, F1, perplexity, a judge score. Fast, cheap, computable on every change, and furthest from anything that matters.
System metrics. End-to-end quality of the full pipeline on realistic inputs: task success rate, groundedness, schema validity. Closer to reality and still offline.
Product metrics. Observable user behaviour: acceptance rate, edit rate, retry rate, task completion time, feature usage. Real signal, available only in production, with a feedback delay measured in days.
Business metrics. Retention, conversion, revenue, cost saved. What the work is for, and moved by many things besides the model, with a feedback delay in weeks or months and enough noise that most changes cannot be detected in them individually.
Validating the links
The useful discipline is to check the correlation between adjacent rungs rather than assume it, once, with real data. Collect a set of changes for which both a model metric and a product metric were measured, and look at whether they moved together.
Teams that do this frequently find the correlation is weak, and the finding is valuable. It means the offline metric is not measuring what the product needs, which is a fixable problem: the metric can be changed, weighted, or replaced by one that tracks. Continuing to optimise a metric that does not correlate with the product outcome is how a team can work hard for a quarter and move nothing.
The choice of which rung to make the primary decision metric follows from feedback speed and reliability. Model metrics gate every commit. System metrics gate deployment. Product metrics decide rollouts. Business metrics validate the strategy quarterly, and using them for individual feature decisions is impossible because they are too noisy and too slow.
When it breaks
Guardrail metrics are essential and usually absent. A change that improves the target metric while degrading latency, cost or a subgroup's experience is a regression sold as an improvement. Every experiment needs metrics that must not degrade alongside the one that should improve.
Aggregate improvement hides distributional harm. A change improving the mean by degrading a segment is common, and it is invisible without segment-level reporting on every experiment.
Long-term effects invert short-term ones. Engagement-optimising changes can improve weekly metrics and degrade retention, and the horizon over which a metric is measured is a choice that determines its answer. Holdback groups maintained over months are the mechanism for catching this.
Novelty inflates early results. A new feature's usage in the first two weeks includes exploration that will not persist, so an early positive result needs a longer measurement before it is believed.
12 flashcards for this concept
Click a card to reveal the answer.