Inference Request Path
The sequence of stages an LLM application request passes through, each with distinct latency, cost and failure characteristics.
An LLM application is not a wrapper around a model call. The request path typically runs: input validation and injection screening → context assembly (retrieval, history, user state) → prompt construction → model invocation → output validation → post-processing → response.
Each stage has different properties, and treating the whole thing as one opaque call is why these systems are hard to operate.
Latency is dominated by the model call, and specifically by output token count — generation is sequential, so a 500-token answer takes roughly five times as long as a 100-token one. Retrieval is usually tens of milliseconds against seconds of generation.
Cost is per token, in both directions. Context assembly decisions therefore have a direct, measurable price: adding five retrieved documents to every prompt is a permanent cost increase on every request.
Failure modes are unfamiliar. The model can be slow, rate limited, or return something well-formed and wrong. Timeouts must accommodate generation length; retries must account for non-determinism; and output validation is not optional, because the model is an untrusted component in the sense that matters.
Streaming changes the architecture rather than decorating it: time-to-first-token becomes the latency metric users experience, and the response path must support incremental delivery end to end.