Inference Telemetry
Recording the full context of each model interaction — inputs, outputs, tokens, latency, model version and evaluation scores — so quality and cost can be investigated.
Conventional observability answers whether the system responded. For an LLM application the harder question is whether it responded well, and that requires recording things a normal service would not.
Per interaction: the prompt as actually sent (after template rendering and context assembly, which is the only version that explains the output), the retrieved context and its sources, the raw output, the model and version, token counts in and out, latency including time-to-first-token, cost, and any evaluation scores.
The aggregate signals worth alerting on: cost per request trending up, which usually means context growth nobody decided on; token usage anomalies, which catch runaway loops early; latency distribution by model; error and refusal rates; and evaluation scores over time, which is how a silent quality regression is detected.
Two things that make this different from ordinary telemetry:
Prompts and outputs frequently contain personal data, so retention, redaction and access control are compliance concerns, not preferences.
A provider can change a model behind the same name. Recording the version and monitoring quality continuously is the only way to detect a regression you did not cause and were not told about.