When the monitor shares the outage
What happens to telemetry, alerting, status pages and incident tooling when they depend on the system they watch, reconstructed from eleven public incidents, the Prometheus Watchdog ecosystem, two Google and Facebook telemetry papers and two practitioner talks.
Sorts eleven public incidents into three failure classes (shared fate, silent pipeline, blind by design), draws the three failure domains of a monitoring architecture, and sets out five decisions with the condition that flips each. A reader can map the dependencies of their own monitoring and response path, say who hears when it goes silent, and run the drill that proves it.
The most recommended safeguard, an always-firing Watchdog watched from outside, is a one-line rule with no native sender in Alertmanager, a feature request in the operator, and two closed-unmerged pull requests around its definition and labelling.
What you get out of it
- Detection and diagnosis fail separately: Google was alerted two minutes into the June 2019 incident and still found debugging 'significantly hampered' because its tools shared the congested network.
- Google's Monarch runs a 'meta' deployment on a proven-stable older version to watch the others, so the watcher shares the fewest release-time failure modes with the watched.
- Slack's external monitor was the detector that survived its January 2021 outage; its internal dashboards and alerting did not.
- Independence is not free of side effects: Cloudflare's off-platform status page went down at the same time as its November 2025 incident and was first read as an attack.
- No incident in the corpus credits a dead man's switch with catching a monitoring outage; whether the pattern works quietly or is rarely deployed is open.
Scope
Why this, now. Public postmortems from 2020 to 2026 keep reporting the same loss of dashboards and alerting mid-incident, and in October 2026 the reference Prometheus stack still leaves the external half of the dead man's switch to the operator.
What it does not cover. SLO design, alert fatigue, tracing, general vendor selection and security monitoring; Facebook's outage duration, Google's own Firebase postmortem and talk video timestamps were not verified.
Other field guides
The cluster nobody finished emptying: ten years of GitHub
A single-company dig built from eight GitHub postmortems and availability reports, four design posts that name a rejected option, and first-hand git …
35 sources · 10 organisations · 8 postmortemsThe delete lands one level up, on the path nobody guarded
Reconstructs the safe-delete architecture from the Atlassian 2022, Azure DevOps 2023, Google Cloud GCVE 2024, Cloudflare R2 2025, AWS S3 2017, GitLab…
34 sources · 21 organisations · 7 postmortemsEvacuating the region: who can actually leave, and what it costs
Reconstructs regional evacuation from GitLab's public DR blueprints, runbooks and gameday timing ledger, Meta's Maelstrom and Taiji papers, Netflix's…
26 sources · 15 organisations · 6 postmortems