An AI feature launched two months ago now costs more per month than the rest of the platform. What do you investigate?

Get cost per request, decomposed

Token cost splits into input and output, and they price differently. Break the bill down by feature, by user, and by input versus output tokens. The distribution is usually extreme, and the answer normally appears immediately.

The four findings, in descending frequency

Context bloat. The prompt grows over time — more instructions, more examples, more retrieved chunks, full conversation history resent on every turn. Input tokens are paid on every request, so a prompt that doubled has doubled the bill with no change in traffic.

No caching. Identical or near-identical requests recomputed. Prompt caching for a stable system prefix, and response caching for repeated questions, are frequently large reductions for modest effort.

Model over-specified. The most capable model used for every call, including classification and extraction that a much cheaper model does equally well. Routing by task is usually the single biggest lever.

Retries and loops. An agent looping, or a failure path that retries an expensive call. Check the distribution of calls per user action — a long tail means something is looping.

The structural fixes

Set a token budget per request type and alert when it is exceeded, the same way a latency budget works. Without one, context grows silently because each addition is individually small.

Route by task, with the capable model reserved for what needs it, and an evaluation set proving the cheaper model is adequate for the rest.

Cap conversation history rather than resending it whole, summarising older turns.

Attribute cost per feature and per team, visibly, so the team that added the examples sees the consequence.

What to tell the business

Unit economics: cost per user action, and what it becomes at ten times the volume. An AI feature that is affordable at current traffic and unaffordable at target scale is a decision to make now rather than at scale — and the answer is usually a smaller model plus better retrieval rather than abandoning the feature.