intermediate 3 min answer Multiple choice

A game day for a payments platform needs 8 engineers for 4 hours, plus about 2 days of one engineer's preparation. The service earns roughly 40000 currency units an hour and has had 3 severity-one incidents in the last year averaging 90 minutes of full outage. Roughly what does the exercise cost, and at what point does it pay for itself?

game daysestimationbusiness casemttrincident cost
Pick one
Show the full answer Hide the answer

The assumptions, stated

  • 8 engineers × 4 hours = 32 engineer-hours for the exercise.
  • Preparation: 2 days of one engineer ≈ 16 engineer-hours.
  • Follow-up actions from findings: assume 1 to 2 engineer-days, which is part of the value rather than overhead, so exclude it from the cost of the exercise itself.
  • Total: about 48 to 50 engineer-hours per game day, run quarterly.

The arithmetic for payback

Revenue at risk: 40,000 per hour, so about 667 per minute.

Current exposure: 3 severity-one incidents a year × 90 minutes = 270 minutes of full outage a year, roughly 180,000 in lost revenue, before any cost of remediation, support load or credits.

Cost of the exercise: 50 engineer-hours. At a loaded cost in the region of 60 to 100 per hour, that is roughly 3,000 to 5,000 per game day, or 12,000 to 20,000 a year run quarterly.

Break-even in minutes of outage avoided per year: 12,000 to 20,000 divided by 667 per minute = about 18 to 30 minutes a year. Against a baseline of 270 minutes, that is a 7 to 11% improvement in annual outage duration — a bar that one clear finding about where a runbook is wrong clears on its own.

Which assumption dominates the error

Revenue per minute. It is the term that varies by orders of magnitude between businesses, and it is the one to get from finance rather than estimate. For a payments platform it also understates the loss, because a failed payment is often not retried: the sale is gone rather than delayed.

The second-largest is whether the exercise changes anything. A game day that produces findings nobody actions costs 50 engineer-hours and returns nothing, which is the most common failure and the reason the follow-up budget should be committed before the exercise is scheduled.

What the number rules in and out

It rules in quarterly game days for any service with material revenue per minute, without a debate, because the break-even is a fraction of the existing exposure.

It rules out the argument that game days must prevent incidents to be worth running. Shortening them is enough, and shortening is the realistic benefit: the exercise's reliable output is a corrected runbook, a discovered missing permission, a dashboard that does not load, and a team that has met the failure before.

It also rules out running them for low-stakes internal services on this reasoning. An internal tool with no revenue per minute needs a different justification, which is usually about the dependency chain rather than about its own downtime.

When not to run one

For an internal service with no measurable revenue per minute and no external dependency on it, the arithmetic above returns nothing and the exercise needs a different justification. Run it then only if the service sits on a chain that something revenue-bearing depends on, and justify it from that dependency's numbers rather than inventing a cost for the tool itself.

Also not yet, if the last few incidents were caused by known unfixed reliability issues. A game day spends 50 engineer-hours discovering weaknesses, and rediscovering ones already on the backlog is the most expensive way to write a ticket.

Why the other options fail

"Only pays for itself if it prevents an incident." This is the framing that gets game days cancelled. Prevention is unprovable and unnecessary; the measurable mechanism is reduced time-to-diagnose, and the break-even in minutes is small.

"About 300 engineer-hours." Off by roughly 6x, usually from counting a full day for all 8 engineers plus a week of preparation. Worth noticing because an inflated estimate is how a cheap practice gets refused.

"Not quantifiable." The most damaging answer available to an architect. Revenue per minute and historical outage duration are both known numbers inside the company, and refusing to combine them cedes the decision to whoever is willing to.