An AI inference service receives computationally expensive requests with unpredictable bursts, where a single request can occupy a GPU for many seconds. Which combination of controls should it use, and in what order do they apply?

Why order matters more than the individual controls

Each control is cheap at one layer and ruinous at another. The organising principle is: reject as early as possible, and never let an expensive resource be consumed by work that will be discarded.

At the edge — authentication, quota, rate limit. Rejecting here costs microseconds. Rejecting after a GPU has begun generating costs seconds of the scarcest resource in the system. Any check that can be made before scheduling must be made before scheduling.

In the queue — priority classes with bounded wait. Not all requests are equal: interactive requests have a user watching, batch requests do not, and a free tier is not a paid tier. A bounded queue with per-class limits gives an honest answer — a request that cannot be served within its class's deadline is rejected now rather than accepted and disappointed later. Fast rejection lets the client fail over or back off; slow rejection wastes both sides' time.

At the accelerator — dynamic batching. This is where the capacity actually comes from. Batching multiple requests into one forward pass raises throughput enormously, and continuous batching, where finished sequences leave the batch and new ones join mid-flight, avoids the whole batch waiting for its longest member. The trade-off is latency for individual requests versus throughput for all, tuned by a maximum batch-formation delay.

Autoscaling — a background control, not a burst control. Acquiring accelerator capacity has a lead time of minutes at best, and often much longer. Autoscaling responds to the trend; it cannot respond to a burst. Treating it as the burst defence is the most common and most expensive design error in this space.

Why the alternatives fail

Autoscaling alone. The burst is over before capacity arrives. Meanwhile every request is accepted, queues grow without bound, latency exceeds every client's timeout, and the system delivers zero useful work while running at full cost.

Unbounded queueing. Sounds generous, is cruel. Requests sit past their deadline; clients time out and retry, adding load; the system computes answers nobody is waiting for. Unbounded queues do not prevent rejection — they convert fast, honest rejection into slow, expensive rejection.

A uniform global rate limit. Either it is low enough to protect the system, and it throttles the largest customers during their normal operation, or it is high enough for them and does not protect anything. Limits must be per-tenant and preferably per-class.

The property that ties it together

The system must always know its admissible rate — how much work it can complete per second — and must refuse anything beyond it immediately. A system that accepts work it cannot complete is not being generous; it is converting a capacity problem into a latency problem and then into a reliability problem.