advanced 3 min answer

A game hosts a scheduled in-world event for 30 million concurrent players. How should matchmaking, instance allocation, pre-scaling, login queues and degradation be planned, and what is rehearsed?

epic-gamesfortnitelive-eventsmatchmakingpre-scaling
Show the full answer Hide the answer

What makes this harder than ordinary peak traffic

  • Everyone arrives in the same few minutes, because the event has an announced start time.
  • The session is stateful and long-lived. Unlike a web request, a player occupies a game instance for the duration, so capacity is measured in concurrent sessions and cannot be recycled quickly.
  • The experience is synchronous and shared — players must be in the event together, so the system cannot spread arrivals over time without spoiling the product.
  • Failure is unrecoverable. A player who cannot join a one-time event has missed it permanently, which changes the acceptable failure rate dramatically.

Pre-scaling and instance allocation

  • Provision to the upper bound hours in advance, warmed and health-checked. Not autoscaling, for the usual reason: the spike is faster than the loop.
  • Pre-allocate game instances before players arrive, so joining is an assignment rather than a provisioning operation. Instance startup on the critical path at this arrival rate is fatal.
  • Regional pre-allocation from expected regional demand, since players must be matched to nearby servers for latency and capacity is not fungible across regions.
  • Verify the raw capacity exists with providers, including in the specific regions.

The login queue is the essential mechanism

At this arrival rate, the authentication and session-establishment path saturates long before the game servers do. A queue at the front:

  • Admits players at a rate the backend can absorb, converting a failure into a wait.
  • Must communicate honestly — position and estimated time — because an opaque queue is indistinguishable from a hang and generates the same support load as an outage.
  • Must be fair, typically first-come-first-served with a token, and must be resistant to being bypassed by clients that retry aggressively.
  • Must hold a place across a disconnection, or a network blip sends a player to the back and the queue becomes its own failure mode.

Matchmaking under load

  • Relax quality constraints as load rises. Skill-based matching is a luxury when the objective is to place 30 million people into instances quickly; a worse match now beats a better match after the event has started.
  • Pre-form groups where possible, so friends are placed together in one operation rather than through repeated re-matching.
  • Simplify to assignment for a one-time event. For a shared world event, matchmaking may reduce to filling instances in order, which is dramatically cheaper.

Degradation

Non-critical services shed first, in a pre-agreed order: the item shop, cosmetics and lockers, social feeds, leaderboards, statistics recording, replays, achievements and progression writes. Progression can be buffered and applied afterwards rather than written synchronously — a large saving.

The event itself and the ability to join never degrade, because they are the product.

What is rehearsed

  • A full-scale load test against production capacity, above the expected number.
  • A dry run of the event with internal and invited players, exercising the real path end to end.
  • The queue mechanism under real arrival rates, since a queue that fails under load is a spectacular failure.
  • The degradation switches, each thrown in production.
  • The incident command structure, with a named decision-maker who can shed features without an approval chain.
  • A change freeze in the preceding days.

The distinguishing property of these events is that the date is known, which converts an unpredictable scaling problem into a planning problem. Organisations that fail scheduled events usually failed at planning, not at engineering.