Observability for LLM Applications
Why the classical monitoring stack does not transfer to systems with free-text output, what a trace over an agent must capture, and the online quality signals that work without ground truth.
Classical model monitoring assumes a fixed feature vector, a small output space, and an eventual label. An LLM application has unbounded text input, unbounded text output, no label, several model calls per request, and a model that may have changed underneath you without notice. Almost every assumption the monitoring stack was built on is gone.
What a trace must capture
A single user request may involve retrieval, several generations, tool calls and a final synthesis. The unit of observability is the trace, structured as a tree of spans, and each span records:
The full prompt as sent, including the system prompt, retrieved context and conversation history, since the assembled prompt is the actual input and reconstructing it later from its components is unreliable. The model and its parameters, with the provider's model identifier and version. Token counts for input and output, which are the cost and the latency. The output, and for tool calls the arguments and the result. Latency broken into time to first token and generation time, because they have different causes and different user impact.
The assembled prompt is the field that most often gets omitted for volume reasons and is the one without which nothing can be diagnosed.
Quality signals without ground truth
Implicit user feedback is the most reliable signal available and is usually under-instrumented. Whether the user accepted the suggestion, edited it, retried, rephrased, or abandoned the session are all evaluations of the output. A retry rate is a quality metric with no annotation cost.
LLM-as-judge on a sample scores outputs against a rubric. It scales, it correlates moderately with human judgement, and its biases are documented: preference for longer responses, position effects in pairwise comparison, and a tendency to favour text stylistically similar to its own. It is usable for detecting change over time and unreliable as an absolute measure, so tracking the judge's score as a time series against a fixed prompt and fixed judge version is the right use.
Automated checks cover what is checkable: output schema validation, refusal detection, citation presence, whether claims appear in the retrieved context. Groundedness checking, verifying that assertions are supported by the provided sources, is the highest-value automated check for a retrieval system.
Retrieval quality is separately measurable and separately fixable, and a large fraction of generation failures are retrieval failures wearing a different costume. Logging retrieved documents and their scores lets that be diagnosed rather than guessed.
When it breaks
The model changes without you. A hosted model updated behind a stable name changes behaviour with no deployment on your side. Pinning to dated versions where the provider offers them, and running a fixed evaluation set on a schedule, is the only way to notice. Attributing a quality change to your prompt when the model moved is a common and expensive misdiagnosis.
Cost is dominated by a small share of requests. Token usage is heavy-tailed: a small fraction of requests with long contexts or long agent loops accounts for most spend. Monitoring mean tokens per request hides this completely, and the useful views are the p95 and p99 plus cost attributed per feature and per user.
Logging prompts logs user data. Prompts contain whatever users typed, which is unpredictable and frequently sensitive. Retention, redaction and access control on trace storage are a privacy problem of a different shape from feature logging, because the content is unstructured and cannot be classified in advance.
Agent traces are deep and hard to read. A twenty-step agent run produces a trace nobody inspects manually. Aggregate views, step counts, tool error rates, loop detection, and the distribution of trajectory lengths, are what make agent behaviour observable, and per-trace inspection is a debugging tool rather than a monitoring one.
Evaluation drift is invisible without a fixed set. If the evaluation prompts, the judge model, or the judge prompt change, the score time series is not comparable across the change. Versioning the evaluation harness as strictly as the application is what keeps the metric a measurement rather than a coincidence.
14 flashcards for this concept
Click a card to reveal the answer.