A device-testing platform receives a burst of tens of thousands of test sessions after a major framework release. How should queues, worker pools, concurrency limits, resource classes, scheduling fairness and backpressure prevent capacity exhaustion?

The structure of the problem

Sessions are long-running, resource-intensive, and heterogeneous — a two-minute unit test and a forty-minute end-to-end suite compete for the same devices. Without structure, a few large customers consume everything and the platform appears down to everyone else.

The controls, and what each one does

• Per-customer concurrency limits, derived from their plan. This is the primary fairness mechanism and it is a product decision as much as a technical one. It bounds any single customer's consumption regardless of how many sessions they submit.
• Per-customer queues, scheduled fairly, rather than one global FIFO. A global queue means a customer who submits ten thousand sessions delays every customer who submits ten. Weighted fair queueing by plan tier is the usual shape.
• Resource classes with separate pools. Different browser and device types are not interchangeable, so they are separate capacity pools with separate queues — a shortage of one device class should not appear as a general platform failure.
• Admission control at submission. When the queue for a resource class exceeds a depth that implies an unacceptable wait, reject or warn at submission rather than accepting work you cannot start. Accepting everything and queueing indefinitely is the failure mode, because it converts a capacity problem into a wait-time problem that customers discover only after committing.
• Session time limits and idle timeouts. A hung session holds a device forever, and at scale a small percentage of hung sessions removes a large fraction of capacity. Enforced termination is not optional.
• Backpressure to the client — a queue-position API, an estimated start time — so a customer's own CI can decide to wait or fail fast rather than blocking a pipeline blindly.

The scheduling detail that matters most

Do not start a session you cannot finish. Because sessions are long, starting one on capacity that is about to be reclaimed wastes the whole session rather than a fraction of it. Admission should reserve for the expected duration, which means duration estimation becomes part of the scheduler.

Where the elasticity comes from

Burst capacity is expensive to hold idle, so the mature design combines a warm baseline sized to typical load, a pool that scales on queue depth, and — for the cases where waiting is genuinely acceptable — a lower priority class that runs on cheaper preemptible capacity. That gives three tiers of cost and three tiers of latency, and lets the customer choose.