Game Days

Definition

A game day is a planned exercise in which a realistic failure is introduced — injected, simulated, or described — and the team responds using their real tools and procedures. The subject under test is the socio-technical system: detection, communication, decision-making and execution, not just the software.

Why they find things nothing else does

Chaos experiments test whether the system degrades correctly. Game days test whether the organisation can respond, which is a different and usually weaker capability. They reliably surface:

• Runbooks that are wrong, out of date, or stored in the system that is down.
• Missing access — the person on call cannot perform the action required.
• Alerts that did not fire, or fired to a rotation nobody watches.
• Dashboards that do not show what is needed during this specific failure.
• Communication confusion: who declares an incident, who talks to customers, who decides.
• The realisation that failing over requires an approval from someone unreachable at 3am.

Running one well

• Announce that a game day is happening, even if the specific scenario is unknown. Surprise exercises test panic, which teaches little and costs trust.
• Have a facilitator who knows the scenario and can inject information, answer "what do you see", and stop the exercise.
• Use real tools and real channels. An exercise conducted in a meeting room with a whiteboard finds none of the tooling problems.
• Include the non-engineering participants: support, communications, and whoever approves customer-facing statements.
• Time-box, then debrief immediately while the experience is fresh.
• Track findings as work items with owners. A game day that produces a list nobody actions has entertained people.

Industry example

Large operators run these as a standing discipline — Amazon and Microsoft both practise scheduled resilience exercises across services, and the recurring finding across the industry is the same one: the control plane fails the test. Organisations discover during rehearsal that their identity provider, secrets store, deployment pipeline, or observability stack is single-region, and therefore that their carefully-designed regional failover cannot actually be executed.

That is a finding that no amount of architecture review produces, because on paper each component is fine. It only appears when someone tries to use them together under the conditions of the failure.

Failure scenarios

• Scenarios too easy, so everyone passes and nothing is learned.
• Only engineers involved, missing the coordination failures that dominate real incidents.
• Findings not tracked, so the same problems recur next year.
• Run once, so the learning decays as people and systems change.
• Blame in the debrief, which guarantees the next exercise is performed rather than attempted.

Interview question

"Design a game day for a service you support. What is the scenario, who is in the room, and what would count as a successful outcome?"