An AI platform serves computationally expensive requests with unpredictable bursts, where one request can occupy an accelerator for seconds. What is the architectural shape, and which control is most often misplaced?

The shape

Admission control at the edge → priority queues with bounded wait → dynamic batching at the accelerator → autoscaling as a slow background control.

The organising principle is economic: never let a scarce, expensive resource be consumed by work that will be discarded. Rejecting at the gateway costs microseconds; rejecting after generation has begun costs seconds of the scarcest resource in the system.

At the edge: authentication, quota and rate limiting, denominated in the resource that is actually scarce — tokens or a cost-weighted unit — because request counts are meaningless when one request can be a thousand times heavier than another.

In the queue: priority classes with bounded wait. A request that cannot meet its class deadline is rejected now. Fast rejection lets a client fail over or degrade; slow rejection wastes both sides' time and produces work nobody is waiting for.

At the accelerator: continuous batching, where finished sequences leave the batch and new ones join mid-flight, so a long generation does not hold the whole batch. This is where the capacity actually comes from.

Autoscaling responds to trends. Accelerator capacity has a lead time of minutes at best, so treating it as the burst defence is the most common and most expensive design error in this space.

The control most often misplaced

Autoscaling, treated as burst protection. The burst is over before capacity arrives; meanwhile every request is accepted, queues grow without bound, latency exceeds every client's timeout, and the system delivers zero useful work at full cost.

Unbounded queueing is not generosity. It converts fast, honest rejection into slow, expensive rejection.

The capacity property that ties it together

The system must always know its admissible rate — how much work it can complete per second — and refuse anything beyond it immediately. A system accepting work it cannot complete is converting a capacity problem into a latency problem and then into a reliability problem.

And because queueing delay grows with the variance of service time as well as its mean, separating request classes by expected cost improves latency more than adding capacity does — which is usually the highest- leverage change available and costs nothing in hardware.