Inference Cost per Interaction
The fully loaded token cost of one user interaction, including retrieval, retries, guardrails and agent loops, measured rather than estimated.
The unit economics of AI features are frequently modelled from a single prompt and response, and the real cost is several times that, because the visible call is a fraction of what the interaction consumes.
The components that get omitted. Every turn of a conversation resends the accumulated history, so cost per turn grows through a session rather than staying flat. Retrieved context is usually the largest share of input tokens. Guardrail checks and LLM-based evaluation are additional inference calls. Agent loops make several calls per user request, and a loop that goes wrong makes many more. Retries after validation failure double the cost of the requests that need them. And reasoning tokens on models that produce them are billed and invisible in the response.
Measuring it properly means instrumenting per interaction rather than per API call, attributing every call in the trace back to the originating user action. That instrumentation belongs in the AI gateway.
The reason it matters commercially: an AI feature is usually priced or budgeted against an assumed cost per user, and the difference between the modelled and actual figure is what determines whether the feature is viable at scale. It is far better to discover a five-fold discrepancy during a pilot than after committing to a pricing model, and the ceiling controls — token limits, loop bounds, history truncation — are much easier to add before launch than after users have grown used to their absence.