Unit Economics of an AI Feature
How to build a cost-per-request figure that survives scrutiny, why the marginal cost of an LLM feature does not fall with scale the way software's does, and the retry and failure multipliers everyone forgets.
Traditional software has near-zero marginal cost: the tenth million user costs almost nothing more than the ninth. An LLM feature does not work this way. Every request consumes compute proportional to its token count, so cost scales with usage roughly linearly, and a product priced on software intuitions can have negative gross margin at scale while looking fine in a pilot.
Building the number
Start with the token accounting for one request, and count all of it. The input includes the system prompt, retrieved context, conversation history, tool definitions and the user's message. The output includes reasoning tokens where the model produces them, which are billed and are frequently several times the visible answer.
Then apply the multipliers that a naive calculation omits.
Retries. Failed generations, schema validation failures and timeouts each consume tokens and produce nothing. A five percent retry rate is a five percent cost increase on a good day.
Multi-step flows. An agent that averages six model calls per user request costs six times what the single-call estimate suggests, and the variance is high because the step count is data-dependent.
Guardrails and evaluation. Classification of inputs and outputs, safety checks and any LLM-as-judge scoring are additional calls, often to a smaller model, and they are consistently left out of the estimate.
Non-inference cost. Vector database queries, embedding generation, retrieval, storage and the serving infrastructure around the model. For a retrieval-heavy application these can rival the generation cost.
The output is a cost per request that can be multiplied by expected volume and compared against revenue per user. Doing this before launch rather than after is the difference between a pricing model and a discovery.
Where the leverage is
Cost per request is dominated by tokens, and tokens are dominated by input in most retrieval and agent systems. So the highest-leverage reductions are usually about context rather than about model choice: retrieving five documents rather than twenty, truncating conversation history, and removing the accumulated instructions in a system prompt that nobody has audited.
Prompt caching changes this arithmetic substantially where a long prefix is shared across requests, since cached input tokens are billed at a fraction of the standard rate. Designing the prompt so the stable part comes first, and is genuinely byte-identical between requests, is what makes the cache hit.
Routing simple requests to a smaller model is the other large lever, and it requires a classifier whose own cost and error rate enter the calculation.
When it breaks
Averages hide the tail. Token usage is heavy-tailed, so mean cost per request understates the bill and misprices the heavy users. Cost at p95 and p99, and total cost attributed per user, are what reveal whether a small group is consuming a disproportionate share.
Provider pricing moves. Per-token prices have fallen substantially and repeatedly, and model capabilities at a given price point have improved. A unit economic model built on today's prices should be revisited on a schedule rather than treated as a fixed input.
Free tiers and trials are where the loss lives. An unlimited free tier on a feature with real marginal cost is an uncapped liability, and the usage distribution of free users is heavier-tailed than that of paying ones.
Self-hosting changes the shape, not necessarily the total. Running an open-weights model on reserved hardware converts a variable cost into a fixed one, which is cheaper above a utilisation threshold and more expensive below it, and adds engineering cost that belongs in the comparison.
12 flashcards for this concept
Click a card to reveal the answer.