A ticketing platform expects 2 million buyers for a concert on-sale and receives 14 million requests, many from bots. How should waiting rooms, admission tokens, bot mitigation, fair queueing, inventory holds and user-facing status interact?
Show the full answer Hide the answer
The structural problem, stated honestly
Demand exceeds supply by an order of magnitude and no architecture changes that. 14 million requests for a finite number of seats means the overwhelming majority of people will not get one, and the system's job is not to satisfy them — it is to allocate fairly, communicate honestly, and remain available while doing so.
The 2022 Eras Tour on-sale is the most-analysed instance: arrival volume many multiples above forecast, heavily contaminated with automated traffic, against small fixed inventory. The technical failure and the communication failure compounded each other, and the second did more reputational damage than the first.
The layers, in order
1. Bot mitigation, before the queue. This must come first, because a waiting room that admits bots is a fairness mechanism allocating fairness to bots. Device attestation, behavioural signals, proof-of-work challenges, account history and payment-method history — with the accepted reality that mitigation is probabilistic and adversarial, so the design must degrade sensibly when it is wrong in both directions.
2. The virtual waiting room, on infrastructure entirely separate from the ticketing system — typically static assets plus a lightweight token service. It must survive load the main system cannot, because a queue that fails under load is a spectacular and public failure. It issues a token at first arrival, holds the place across disconnection, and admits at a rate derived from measured downstream capacity.
3. Fair ordering. Random assignment among those present at open is defensible and reduces the advantage of fast connections; strict first-come rewards network latency and automation. Whichever is chosen must be stated publicly, because perceived unfairness is the durable damage.
4. Admission tokens: signed, single-use, time-bounded, bound to the session. A place that can be shared or sold recreates the problem the queue solved.
5. Inventory holds with a short expiry — a seat held for a few minutes during checkout, released automatically. Too long and inventory is locked by abandoned carts; too short and genuine buyers lose seats mid-payment. This is a tuning decision with direct commercial consequences and belongs to the business.
6. Honest user-facing status. Position, movement, and an explicit statement when inventory is exhausted.
The failure that does the most damage
Queueing people for an hour and then telling them there is nothing left. That is worse than an immediate refusal, because the wait created an expectation the system could not honour.
The system knows the remaining inventory and the number of people ahead. Once those cross, the honest action is to tell people waiting that they will not get tickets — and this is a product decision that engineering must surface, because commercially it is uncomfortable and technically it is straightforward.
What should have been different at the planning level
- Capacity planned for arrival rate, not for inventory. Fourteen million people arriving is a load problem independent of there being 50,000 seats.
- Forecast treated as unreliable, with the system designed to hold a much larger queue than expected.
- Load tested at the arrival rate, including the bot traffic profile.
- Communication prepared in advance, including the message for "everyone still waiting will not get tickets."
- A commercial decision about staged on-sales, spreading demand across time — an architectural constraint resolved by a business mechanism, which is frequently the cheapest available answer and is rarely considered by engineers.