An inference platform's cost per request varies by two orders of magnitude between requests. What does that imply for pricing, capacity and engineering priorities?

What it implies for pricing

An average cost per request is meaningless as a pricing basis. Charging a flat rate per request means cheap requests subsidise expensive ones, which invites customers to send only expensive ones and makes the economics adversarial.

Pricing must track the actual cost driver — tokens processed and generated — so revenue scales with consumption. This is also why rate limits must be denominated in the same unit: "1000 requests per minute" protects nothing when one request can be a thousand times heavier.

What it implies for capacity

Variance, not just mean, drives queueing delay. A workload whose requests differ by orders of magnitude queues far worse at the same utilisation than a uniform one, and a long request occupying an accelerator blocks shorter ones behind it.

Two consequences:

• Lower utilisation targets than a uniform workload would justify.
• Separating request classes by expected cost into different queues or capacity pools improves latency more than adding capacity does — and costs nothing in hardware. This is usually the highest-leverage change available.

What it implies for engineering priorities

Rank optimisations by total cost contribution, which is cost per request times volume — not by cost per request alone. The expensive tail may be a small fraction of spend; the cheap majority may dominate it. Optimising the most expensive request type is intuitive and frequently wrong.

The highest-leverage levers, in rough order:

1. Batching efficiency, since throughput per accelerator is where the capacity actually comes from. Continuous batching — finished sequences leaving and new ones joining mid-flight — avoids the whole batch waiting for its longest member.
2. Caching repeated work, where the workload permits it.
3. Routing to the cheapest model that meets the quality bar for each request class.
4. Admission control, so no expensive resource is consumed by work that will be discarded — rejecting at the edge costs microseconds, rejecting after generation starts costs seconds of the scarcest resource.

The measurement requirement

Attribute cost per request to customer, request class and model. Without that attribution you cannot price correctly, cannot identify which customers are unprofitable, and cannot tell whether an optimisation helped the traffic that matters.

For a platform where cost variance is this wide, cost attribution is a product requirement, not a finance one — it determines what can be sold and at what margin.