Production Monitoring and Drift Detection

The model passed every offline eval. It shipped. Three weeks later, support tickets spike, and you discover a quiet quality regression that started the day a new prompt template rolled out. This is the "silent regression" failure mode - the model never errors, never refuses, just degrades. Production monitoring exists to make the silent loud.

Three things drift

Covariate shift means the inputs changed but the correct mapping is the same. Concept drift means the correct mapping itself changed. Both are real; the response is different. Covariate shift may mean you need to retrain or extend coverage; concept drift means stale knowledge and likely a RAG or fine-tuning update.

Input drift metrics that earn their keep

• Population Stability Index (PSI). Bin a feature distribution (token length, topic cluster id) at training time and at serving time, then compute PSI between them. Rule of thumb: under 0.1 is stable, 0.1-0.25 is moderate drift, over 0.25 is significant.
• KL divergence over embedding clusters. Cluster training inputs into 50-200 buckets in embedding space, then track the KL divergence between the training bucket distribution and the rolling production bucket distribution. Catches semantic drift PSI on raw features misses.
• New-token rate. Fraction of input tokens that did not appear in training. Spikes signal a new product, new event, or new language users are bringing in.

Output monitoring

• Rule-based validators. For structured outputs (JSON, SQL, function calls), parse rate and schema-validity rate are the cheapest leading indicators. A 2% drop in JSON parse rate is your earliest warning.
• Quality sampling. Sample 1-5% of production responses, score with an LLM judge using your custom rubric, alert on rolling-window quality drops.
• Response-shape metrics. Mean output length, refusal rate, language distribution, latency percentiles. Each one drifting independently flags a different upstream cause.

Shadow deployment and canary rollout

Before any model swap touches user traffic:

1. Shadow. New model receives a copy of production traffic, responses are logged but discarded. You compare distributions and run offline quality scoring against the live baseline.
2. Canary. New model serves 1-5% of real traffic. Watch error rates, latency, and quality metrics on the canary slice vs the baseline slice.
3. Gradual rollout. Step the new model from 5% to 25% to 50% to 100% over hours or days, with automatic rollback wired to alert thresholds.

Big-bang model swaps are how silent regressions reach 100% of users in one deploy.

A/B testing LLMs properly

Per-user randomisation is the only correct unit. Per-request randomisation contaminates results because the same user can see both models in one session and adapt their behaviour. Beyond that:

• Stratify by user segment. Quality wins on power users do not imply quality wins on first-time users; report both.
• Latency-correct the metrics. A slower model can lose on engagement metrics for purely UX reasons even if its outputs are better. Either match latency budgets across arms, or include latency as a covariate.
• Pre-register the primary metric. If you measure ten metrics post-hoc and pick the favourable one, you are running a fishing expedition, not an experiment.
• Pick a horizon. LLM quality wins often compound over weeks as users learn what the new model can do. A 24-hour A/B can mislead in both directions.

The silent regression failure mode

The dangerous regression is the one that does not error: outputs become subtly less helpful, slightly more verbose, marginally more hedged. None of your error metrics move. The detection path is a combination of:

• Rolling LLM-judge quality scores on a sampled slice, with weekly trend alerting.
• A stable held-out eval set that runs against production traffic shadowing on a schedule.
• User-facing thumbs-up/thumbs-down or implicit-feedback signals fed back into the dashboard.

Without at least two of these, silent regressions reach you via the support queue first, which is the most expensive notification channel you have.

Trade-offs

• Monitoring adds cost. Sampling 5% of production for LLM-judge scoring is non-trivial budget. Tune the sample rate to the cost-vs-detection-latency frontier you can afford.
• Drift alerts have false positives. Holiday traffic, marketing campaigns, and seasonal cycles all move input distributions without indicating a problem. Tune thresholds to your seasonality.
• Embedding-based drift detection inherits embedding-model bias. If your embedding model under-represents a new user segment, drift on that segment will look small even when it is large.

Further reading

• HELM holistic evaluation methodology - Stanford CRFM
• lm-evaluation-harness for reproducible offline eval
• OWASP LLM Top 10 - covers unbounded consumption and output handling
• NIST AI Risk Management Framework