Reliability & Operations 18 Oct 2026 36 min read

When the monitor shares the outage

What happens to telemetry, alerting, status pages and incident tooling when they depend on the system they watch, reconstructed from eleven public incidents, the Prometheus Watchdog ecosystem, two Google and Facebook telemetry papers and two practitioner talks.

Sorts eleven public incidents into three failure classes (shared fate, silent pipeline, blind by design), draws the three failure domains of a monitoring architecture, and sets out five decisions with the condition that flips each. A reader can map the dependencies of their own monitoring and response path, say who hears when it goes silent, and run the drill that proves it.

The finding that surprised me

The most recommended safeguard, an always-firing Watchdog watched from outside, is a one-line rule with no native sender in Alertmanager, a feature request in the operator, and two closed-unmerged pull requests around its definition and labelling.

What you get out of it

  • Detection and diagnosis fail separately: Google was alerted two minutes into the June 2019 incident and still found debugging 'significantly hampered' because its tools shared the congested network.
  • Google's Monarch runs a 'meta' deployment on a proven-stable older version to watch the others, so the watcher shares the fewest release-time failure modes with the watched.
  • Slack's external monitor was the detector that survived its January 2021 outage; its internal dashboards and alerting did not.
  • Independence is not free of side effects: Cloudflare's off-platform status page went down at the same time as its November 2025 incident and was first read as an attack.
  • No incident in the corpus credits a dead man's switch with catching a monitoring outage; whether the pattern works quietly or is rarely deployed is open.

Scope

Why this, now. Public postmortems from 2020 to 2026 keep reporting the same loss of dashboards and alerting mid-incident, and in October 2026 the reference Prometheus stack still leaves the external half of the dead man's switch to the operator.

What it does not cover. SLO design, alert fatigue, tracing, general vendor selection and security monitoring; Facebook's outage duration, Google's own Firebase postmortem and talk video timestamps were not verified.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.