Token Cost Attribution
Assigning inference spend to features, tenants and users, so that cost can be managed by the people who influence it.
LLM spend concentrates in a small number of code paths, and without attribution nobody knows which. The usual discovery is that one feature accounts for most of the bill, and frequently it is not the one anyone expected.
Attribution requires recording, per request, the feature or endpoint, the tenant and user, the model, and input and output token counts — then aggregating. The gateway is the natural place, because it sees everything and the applications cannot be relied upon to be consistent.
The levers it makes measurable, in rough order of return:
Model routing — most requests do not need the most capable model, and the cost difference exceeds an order of magnitude.
Context discipline — retrieved documents, conversation history and few-shot examples are all charged on every request. Sending five documents where three suffice is a permanent cost on every call, and context tends to grow silently.
Output length limits, since output tokens are typically several times the price of input tokens and generation dominates latency too.
Caching — exact-match caching for repeated queries, and provider-side prompt caching for a shared prefix, which can cut the cost of a large system prompt substantially.
Batching for asynchronous workloads, where providers offer materially lower rates.
The governance that keeps it managed: per-team cost visibility and budgets, since a central bill that nobody owns only grows.