Six weeks before a high-demand on-sale the load test replays last month's traffic mix at four times the volume and the system passes comfortably. The last on-sale degraded badly at a similar multiple. Design the test that would actually predict the event.
Show the full answer Hide the answer
Requirements that drive the test
Three facts about an on-sale break the assumption that a scaled-up replay is predictive.
The arrival shape is a step, not a ramp. Demand goes from background to peak within a second of the gates opening, so every warm-up, autoscaler reaction time and connection-pool growth curve is skipped. Scaling yesterday's mix by 4x reproduces the volume and not the derivative.
The mix collapses onto one or two endpoints. Access-code validation and inventory hold take the traffic; browse and recommendations do not. A proportional replay spreads load across endpoints that will be idle and under-tests the two that decide the outcome.
The population is adversarial and amplifying. Ticketmaster's own statement on the November 2022 Eras Tour on-sale reports about 3.5 billion total system requests, roughly four times its previous peak, attributing much of it to bot attacks and to fans without codes; it also records that over 3.5 million people pre-registered with about 1.5 million codes sent, and that the company estimated about 15% of site interactions experienced issues, including passcode validation errors. A client that fails and retries generates more load than one that succeeds, so the failure itself is an input to the load.
The design
- Drive by arrival rate, not virtual users. An open model keeps offering requests when the system slows. A closed pool of virtual users waits politely for responses and silently reduces offered load exactly when you need it maintained.
- Three populations, separately parameterised: coded users who will succeed, uncoded users who will be rejected, and automated clients hammering the validation endpoint in a loop with no backoff. Give each its own arrival function and its own retry behaviour.
- Make retry amplification explicit. Model each human as issuing a fresh request every 2 to 3 seconds while in the queue, and each bot an order of magnitude faster. Report the ratio of offered to useful requests; that ratio is the number capacity planning needs.
- Put the waiting room under test, not the storefront. The admission mechanism is the system that decides the event's outcome, so assert on its behaviour: does it hold position fairly, does it shed the right traffic, does it degrade to a static page.
- Measure the neighbours. Run normal catalogue traffic concurrently and assert its latency budget. The on-sale is a noisy-neighbour event and the first customer complaint may come from someone buying a different ticket.
- Assert on admission and correctness, not just p99: zero oversells, bounded hold expiry, and a defined error page rather than a timeout.
What it costs
A generator fleet large enough to produce the step - typically tens of hosts, since one generator saturates on connection setup long before the target does - plus an inventory dataset of comparable size and distribution, plus traffic shading if any of this runs in production so synthetic holds never consume real seats. Budget two engineers for the six weeks, and expect the first two runs to measure the generator rather than the system.
When this is the wrong answer
When the event is under about twice normal peak and the bottleneck is already known. Then a single-endpoint saturation test against the known constraint answers the question in a day, and the elaborate profile is ceremony. The profile earns its cost when the failure mode is amplification you cannot predict from a component test, which is exactly when the multiple is large and the traffic is hostile.