Prompt & Configuration Registry  ·  View 05 of 21  ·  People and journeys

Journey — Rolling Back at 3am

The journey the architecture is actually optimised for, run by someone who does not own the thing they are reverting.

Editable source SVG draw.io All views
On-call engineer does not own the prompt Goal — Make it stop, before understanding why Trigger — Paged: refusal rate on the billing assistant up 9× in twenty minutes Done when — Behaviour back to yesterday's inside a minute, with every bit of evidence still intact 1 · Page 2 · Orient ◆ moment of truth 3 · Decide ◆ moment of truth 4 · Revert 5 · Confirm What they do Opens the alert Asks what changed Reads the digest history Picks last known good Confirms rollback Checks the metric Hands to the owner What the platform does Alert names the key Lists last 10 live digests Shows serving windows Diffs the two versions Pointer write, 2 s Notice + poll backstop 99% converged at 8 s Audit entry written How it feels In control Coping Lost Where it hurts Which of six changes? Is the fleet even uniform? No approver awake Signal lags 10 minutes What answers it Serving windows per digest Skew is a metric, not a guess Rollback needs no approval Same path as promotion Kill switch if it is not it Journey — On-Call Engineer Rolls Back at 3am Orient is the trough, and it is a data problem, not a UX one: the registry must already know which digest served which window. v 1.0 · owner Platform Architecture · d 2026-09

The trough, and what answers it

  • Orient is the bottom of the line: six things changed today and the fleet may not be uniform. The engineer cannot act until they know which digest served the bad window.
  • This is a data requirement, not an interface requirement — the registry must already hold the last ten digests that served production and the window each one served (ask.md §6).
  • Version skew is made a metric rather than a surprise, precisely so this phase has an answer instead of a theory.

Decisions this journey forces

  • Rollback requires no approval. An approval gate on the reverse operation is an availability decision disguised as a control (ADR-04).
  • Rollback is the same mechanism as promotion, run backwards — so the path is exercised every day rather than once, under pressure.
  • The kill switch exists for the case where rollback is not enough: it detaches the key from every experiment and serves the declared safe default in one action.

Residual risk

  • The confirming signal lags the revert by around ten minutes, so the engineer hands over without certainty. Accepted: the alternative is holding traffic on a suspect version until a slow metric agrees.
  • A rollback that is not the fix costs a second cycle. The kill switch bounds that cost, at the price of the feature degrading to its default.