What must be observable in an AI feature that is not covered by conventional application monitoring?

What conventional monitoring covers

Latency, error rate, throughput, saturation. All necessary and all blind to the failure that matters most: the system returns a successful response containing a wrong or unhelpful answer. Every technical signal is green.

What must be added

1. The full interaction trace. For each request: the retrieved context with its sources and scores, the prompt version, the model version, the parameters, the raw output, the tools called with their inputs and outputs, and the final rendered result.

Without this, an incident report — "the assistant gave a customer wrong information" — is uninvestigable. This is the single most important addition and it is frequently omitted because the data is large and sensitive.

2. Quality proxies, continuously. Refusal rate, abstention rate, output length distribution, retrieval score distribution, and the proportion of requests where retrieval returned nothing. Sudden shifts in any of these indicate a change in behaviour that no error rate will show.

3. User-signal metrics. Edit rate on generated content, regeneration rate, thumbs-down rate, escalation to a human, task abandonment. These are the closest available proxy for quality and they respond to regressions within hours.

4. Cost and token accounting per request, attributed to feature, tenant and model. Cost varies by orders of magnitude between requests, so an aggregate is meaningless for both pricing and optimisation.

5. Truncation and context-limit events, which degrade quality silently rather than erroring.

6. Version correlation. Every output tagged with prompt version, model version, retrieval configuration and index version, so a quality change can be attributed to a specific change.

The privacy constraint that shapes it

Interaction traces contain user content, so the observability system inherits the data classification of the most sensitive content it handles. That requires redaction policy, access control commensurate with the data, and retention limits — and it must be designed at the start, because retrofitting redaction into a trace store that already contains months of user content is a much harder problem.

The alerting that works

Alert on distribution shifts and user signals, not on error rates. A model or prompt change that degrades quality produces no errors, no latency change and no saturation — only a shift in output characteristics and in how users respond. Those are the only signals that detect it.