Request one Portfolio ↗
Reliability & operations 12 chapters about 13 min 17 Oct 2026 An original idea

The 3 a.m. page

How an incident platform wakes the right person when everything else is on fire: the whole design, its decisions and its traps, told on a stormy coast.

Start full screen ↗ Play it here ↓

Drag to look around · arrow keys step through · T for the text view

A narrated 3D walkthrough of the Incident Management Platform design: a paging path kept apart from the incident record, one alert followed to a ringing phone, why only an acknowledgement counts, and what each failure does.

The brief it was built from

Walk an engineer or architect through the Incident Management Platform use case: how it works, why it is built this way and what would break it, at the level of the diagram views and decision records, with the stack, the patterns and a hypothetical product interface on screen.

What is real, and what is a scenario

Real data
  • The requirement: availability, latency, throughput, recovery and retention targets, the failure classes and the critical decision (ask.md)
  • The two-plane design, the three paging cells, the four stores and the 32 decision records with the alternatives that lost (docs/decision-record.md, docs/architecture-one-pager.md)
  • The 15 second and 5 second budgets, 20,000 alerts a second for 120 seconds (2.4 million alerts), 72 hours of 15 minute coverage buckets, the sixty second duplicate claim, ten seconds to minority mode (ask.md, views 12, 19, 21, 23, ADR-14, 19, 04)
  • The channel policy for a severity one: push at once, text at thirty seconds, voice at two minutes, inside a five minute step (view 19)
  • The outpost premium of about 55,000 US dollars a year and the planning volume of 60,000 notifications a month (ADR-32, ask.md)
  • Facts about outside products come from the use case's decision record; the Meta outage is described from secondary reports
Scenario & assumptions
  • The responder phone, the status pages and the operator screens are an illustrative interface, not the real product, and carry no brand; the checkout service is an invented example
  • The split of the fifteen second budget (40 and 120 milliseconds, 20 milliseconds of lookups, 800 milliseconds to hand off, about 6.5 seconds in the carrier network and phone) is this walkthrough's own; view 12 only bounds the first stages at 200 milliseconds
  • Five features in a row each at 99.9 percent giving about 3.6 hours of downtime a month is an illustration of series availability, not a measurement
  • The 45 second wait for an unrelated page when dispatch shares cores with ingest is an illustrative number; the design reserves the cores precisely to avoid it
  • The extra 400 milliseconds when carrier A refuses is an illustrative number
  • The metaphor (flares, lighthouses, lit windows) is a teaching device, and the daylight chart maps each part to its real component
  • Meters are illustrative; the advice on what to build first follows the record's MVP list
  • The requirement is a design for an on-premises organisation of 400 responders; the walkthrough does not describe any company's real system

What you leave with

  • A one-sentence essence: the paging path can wake the right person with everything else gone
  • Why the paging path and the incident record are two planes, what the split costs and when not to do it
  • Why only an acknowledgement is proof, and why a duplicate is better than a miss
  • Why on-call is worked out 72 hours ahead and checked by two resolvers
  • What to build first, what it costs, and how to know it is healthy
  • A checklist of questions to ask of your own design

Sources

3D with three.js (MIT) · Type: Bricolage Grotesque and DM Mono (SIL Open Font Licence), Inter · Narration voice: Kokoro-82M v1.0, voice af_heart (kokoro-onnx 0.6.1). Apache-2.0 (Kokoro-82M weights).

Keep going

More walkthroughs

All walkthroughs →