Chaos Engineering Platform  ·  View 05 of 21  ·  People and journeys

Journey — Running the Quarterly Region Game Day

A facilitator measures how long people take to notice and mitigate, which is the output no automated run produces.

Editable source SVG draw.io All views
Reliability engineer facilitator, 40 participants Goal — Measure how long it takes people, not machines, to notice and mitigate Trigger — The quarterly calendar, and last quarter's unclosed findings Done when — Time-to-detect and time-to-mitigate recorded, with the reasons they were slow 1 · Plan 2 weeks out 2 · Brief T-30 min 3 · Run ◆ moment of truth 4 · Escalate unplanned 5 · Stand down verify clean 6 · Debrief same day What they do Sequences 14 experiments Briefs the room Starts the sequence Marks observations Decides to continue Confirms all reverted Publishes the timeline Where Game-day plan Shared timeline Timeline + chat Kill switch panel Settling check Debrief report How it feels In control Fine Exposed Behind the glass Sequencer Notification fan-out Run controller Radius reservation Incident-feed gate Reversion verifier Evidence store Where it hurts Radii collide, order rewritten Two teams page each other A real incident opens One revert does not confirm What would fix it Radius conflicts shown at plan time Annotation on every affected dashboard Automatic suspension on incident open Out-of-band reverter, one click How we know Plan rework 3 rounds Attendance 82% Detect p50 4 min Suspensions 2 per day Stuck reverts 1 per 900 Findings filed 9 User Journey — A Reliability Engineer Runs the Quarterly Region Game Day Time to detect and time to mitigate are the outputs of this view; the technical verdicts are secondary here. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Time to detect and time to mitigate are the primary outputs of a game day. The technical verdicts are secondary here, because the thing under test is the response, not the service.
  • A real incident opening mid-session suspends injection automatically. The facilitator is not asked to make that call under pressure.
  • Stand-down is a verified step, not an assumption: the settling check has to pass before the session closes.

Risks

  • Radius collisions found at run time force the plan to be rewritten live, which is why reservation conflicts must surface at plan time (view 14).
  • One unconfirmed revert per ~900 injected faults — a stated assumption — is enough to make the out-of-band reverter a design requirement rather than a contingency.

Assumptions

  • Fourteen sequenced experiments, 40 participants, 82% attendance, median detection four minutes. Stated assumptions from comparable published game days.
  • Two automatic suspensions a day from the incident feed, which sets how tolerant scheduling has to be of skipped windows.
◀  Journey — Proving a Dependency Is Really Optional All views Layered Architecture  ▶
Solution Architecture v1.0 · Google Cloud · Reliability Architecture · 2026-09 · 21 views · 16 architecture decision records05-journey-run-a-game-day