Design a game day for a payment service. What is the scenario, who is in the room, and what counts as success?

What is being tested

Whether you test the organisation rather than the software, and whether your success criteria measure learning rather than performance.

The scenario

The payment provider's API begins returning intermittent 30-second latencies, then a mixture of timeouts and 500s, without ever going fully down.

Why this scenario:

• Grey failure is harder and more realistic than a clean outage. Systems handle "down" better than "slow and inconsistent", and monitoring usually detects the former and misses the latter.
• It exercises the parts of the system most likely to be wrong: timeout configuration, retry behaviour, circuit breaker thresholds, and the queue that builds up behind it.
• It forces a genuinely difficult business decision: do we queue payments and accept them optimistically, or decline and lose the transactions? That decision cannot be made by engineers alone, which is the point.

Who is in the room

• The on-call engineer, running it as they would a real incident, on their real tools.
• The incident commander.
• Customer support, who will receive the contacts and need to know what to say.
• A product or commercial owner, who must make the queue-or-decline call.
• A facilitator who knows the scenario, injects information, answers "what do you see", and can stop the exercise.
• An observer taking notes — not participating, purely recording timings and friction.

The non-engineering participants matter most. Coordination failures dominate real incidents, and an exercise involving only engineers finds none of them.

What counts as success

Not "the team resolved it quickly". Success is findings, and a smooth exercise that found nothing is a failed exercise.

Specifically, measure:

• Time to detect. Did an alert fire, or did the facilitator have to tell them? This gap is usually the largest and most valuable finding.
• Time to correct diagnosis, and whether the dashboards actually helped.
• Whether the runbook was accurate, findable, and not stored somewhere affected by the failure.
• Whether the person on call had the access to do what was needed, or had to wake someone.
• Whether the business decision was made, by whom, and how long it took to find that person.
• Whether customers would have been told anything, by whom, and when.

Running it well

• Announce that a game day is happening, even if the scenario is secret. Surprise exercises test panic, which teaches little and costs trust.
• Use real tools and real channels. A whiteboard exercise finds no tooling problems.
• Time-box it, then debrief immediately while it is fresh.
• Track every finding as a work item with an owner and a date. A game day producing a list nobody actions has entertained people.

The finding you should expect

Across the industry the recurring discovery is that the control plane fails the test. Teams discover mid-rehearsal that their identity provider, secrets store, deployment pipeline or observability stack has a dependency that makes the planned response impossible.

No architecture review produces that finding, because on paper each component is fine. It appears only when someone tries to use them together under the conditions of the failure.