Eleven public incidents, two Google-scale telemetry papers and the Prometheus Watchdog ecosystem, read for one question: what happens to the thing that tells you the system is broken when it is the system that is broken. A reader leaves able to draw the failure domains of their own monitoring, say who hears about silence, and run the drill that proves it.
The monitor is a service with its own dependencies, and the evidence it fails is often its silence. Who has been bitten, how, and what this page leaves out.
State the problem without the tooling. A system fails. Some other system is supposed to notice, tell a human, and then help that human work out why. If the second system depends on the first for its data, its network, its name resolution, its credentials or its power, then the failure removes the observer in the same instant it creates the thing to be observed. The distinctive difficulty is that the observer's failure usually looks like good news: no alerts, a green page, a quiet pager.
The public record on this is better than the engineering-blog genre would suggest, because outages made the problem visible by accident. Facebook's October 2021 postmortem says the total loss of DNS "broke many of the internal tools we'd normally use to investigate and resolve outages like this" and that even out-of-band access was down [1]. Roblox lost 73 hours and names "a circular dependency between our telemetry systems and Consul" [4]. Slack's dashboarding and alerting service became unavailable while the team was triaging, so "all debugging and investigation was now hampered by the lack of our usual dashboards and alerts" [5]. Datadog's own March 2023 outage began with customer monitors "unavailable and not alerting" [7]. AWS could not update its Service Health Dashboard in 2020 because the posting tool used Cognito, which was itself affected [3].
I sort the thirteen cases here into three classes, and the sorting is mine; the postmortems do not use these names. Shared fate is a monitor that rides the same substrate as the product. Silent pipeline is a monitor whose own path fails without raising anything, because the only evidence of failure is an absence. Blind by design is a monitor that works exactly as built and measures the wrong layer. Each class has a different repair, and teams that fix only the first keep meeting the other two.
The industry's settled answer to the silent pipeline, an always-firing Watchdog alert watched by an external receiver, is a one-line rule in the reference Prometheus stack [13], yet Alertmanager has no native heartbeat sender [16], the operator needed a feature request to ship one [17], and two pull requests that touched how the Watchdog is defined or labelled were closed unmerged [14] [15]. The most recommended safeguard in the field is still, in the reference implementation, an assembly exercise left to the operator.
Scope. This guide covers the failure of the observer: telemetry, rule evaluation, paging, status pages and the tooling humans use during an incident. It does not cover choosing SLOs, alert fatigue, tracing, or vendor selection for monitoring in general, and it treats security monitoring only where the same dependency argument applies. Pricing is included for the independent pieces only.
Three failure domains, a human response path that crosses them, and one component, the heartbeat receiver, whose only job is to treat silence as an event.
Strip the incidents of their brand names and the same shape is left. The product lives in one failure domain, and its shared dependencies (name resolution, identity, configuration, the network itself) sit underneath it. The telemetry plane, meaning collectors, a time-series store, rule evaluation and alert routing, is a second system, and the question in every postmortem above is whether it is a second failure domain or a tenant of the first. The third domain is the one almost nobody draws until after an incident: the components that live outside both, a heartbeat receiver, an external probe, the pager and the status page.
Both Google and Facebook designed their time-series stores around the incident rather than the steady state. Monarch trades consistency for availability because blocking on Spanner "would increase mean-time-to-detection and mean-time-to-mitigation" [31]. Gorilla states "High availability trumps resource efficiency" and keeps two in-memory copies so the store can withstand a regional disaster [32]. Both accept lossy data on the write path as the price of being up.
Monarch has three production deployments. The third, "meta", runs "a proven-stable older version of Monarch and monitors all other Monarch deployments" [31]. Nobody says why the meta tier is old rather than large; I read it as removing the most common shared fate between watcher and watched, which is the same release. That is an inference.
Runs this way at: Google [31]
Grafana's Loki documentation says to collect data about Loki "in a separate instance of Loki, Prometheus, and Grafana" so a broken cluster can be troubleshot "from a working one" [20]. Its Mimir page names the practice meta-monitoring without requiring separation [21], which is a small example of how the advice drifts between sibling products.
The kube-prometheus rule is vector(1), "always firing", labelled severity none, and its text sends the
reader to a mechanism that notifies "when this alert is not firing" [13]. The runbook names the
DeadMansSnitch integration in PagerDuty [22]; Nanit describes the same idea as a vigilance control
[28]; the Prometheus project's own guidance asks for an end-to-end blackbox test of the alert path from outside [19].
When Cognito was affected, AWS posted dashboard updates with "a backup tool that had fewer dependencies, though it was slower and less familiar to support operators" (retrieved summary) [3]. Python moved its status page's DNS to a different provider after its 2016 outage [12]. Both are the same move: shrink the dependency list of the thing humans use when everything else is down.
Mercari built a Terraform module for each step of the flow it wanted to observe, so each stage of the pipeline has an alert policy rather than the pipeline having one global health indicator [24]. Stile's SREcon talk generates absent-alerts for every metric the team expects to exist, because otherwise alerts "aren't going to fire and you won't know that they've gone away" [35].
The human response path deserves its own line in the diagram because it is where Facebook and Google lost time. Facebook's tools broke because DNS broke [1]; Google's tools "compet[ed] over use of the now-congested network" [8]. In neither case was the telemetry the problem. The detection worked (Google was alerted "two minutes after it began"), and the diagnosis stalled because the tools a person needs to follow an alert, a shell, a ticket, a runbook, a chat room, travel on the network that failed.
Five forks. For each: what the sources chose, what they rejected, and the condition under which the rejected option becomes right.
A caution about the evidence before the decisions. These are not five ADRs. No organisation in the corpus published a record that says "we considered X for the monitor and rejected it". The decisions below are reconstructed from postmortem remediations, documentation and issue threads, and each row says which. The conditions in the last column are mine, derived from the stated reasons.
vector(1), shipped in kube-prometheus [13]| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Telemetry location | Separate instance, availability-first store | Same cluster or HA exclusion | Shared substrate removes sight at the worst time | [20] [4] [2] [6] |
| Detector of silence | External heartbeat receiver | Self-check, watcher of watchers | A stopped evaluator cannot report itself | [13] [25] [28] |
| What Watchdog proves | Open question in the chart | Single-stage proof | Stages fail independently | [14] [15] [24] |
| Status page | Separate DNS, off-platform, timestamped | Same platform, server-side only | Shared fate; wrong witness | [12] [10] [11] [27] |
| Response tooling | Fewer-dependency fallback | Standard stack only | Tools degrade with the network | [3] [8] [5] |
Eleven incidents and two smaller cases, grouped by failure class. The design rule in each is this guide's reading of the postmortem's own remediation.
No incident in this corpus credits a dead man's switch with catching a monitoring outage. The nearest case is Slack, where an external monitor paged first [5], but that monitor watched the product, not the telemetry plane. Either the pattern works quietly and nobody writes it up, or it is rarely deployed. The public record cannot separate those readings, and that is the reader's risk.
Durations from the postmortems, prices from vendor pages fetched on 18 October 2026, and one derived comparison with its arithmetic shown.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Outage duration | 73 h | Roblox | Telemetry/Consul circular dependency | 2021 | [4] |
| Time to alert | 2 min | Google Cloud | Network control failure | 2019 | [8] |
| Incident duration | 3 h 19 m to 4 h 25 m | Google Cloud | Same incident | 2019 | [8] |
| Incident duration | 6 h 52 m (derived) | AWS | 7:30 AM to 2:22 PM PST, 7 Dec 2021 | 2021 | [2] |
| CloudWatch impact | 17 h 16 m (derived) | AWS | 5:15 AM to 10:31 PM PST, 25 Nov 2020 | 2020 | [3] |
| Incident duration | 48 h 22 m (derived) | Datadog | 06:03 UTC 8 Mar to 06:25 UTC 10 Mar | 2023 | [7] |
| Incident duration | 40 h 42 m (derived) | Cloudflare | 11:43 UTC 2 Nov to 04:25 UTC 4 Nov | 2023 | [6] |
| Heartbeat service | $0 / $20 / $80 per month | Healthchecks.io | 20 / 100 / 1,000 jobs | 2026-10-18 | [37] |
| Heartbeat add-on | $20 per month per 10 | Better Stack | 10 included; about $2 each beyond (derived) | 2026-10-18 | [38] |
| Status page add-on | $15 per month | Better Stack | One included | 2026-10-18 | [38] |
| Hosted status page | $29 to $1,499 per month | Atlassian Statuspage | Hobby to Enterprise | 2026-10-18 | [39] |
| Downtime cost | >$300,000 per hour | ITIC survey | Over 90% of mid-size and large firms, self-reported | 2024 | [40] |
The comparison that matters is lopsided enough that its precision does not. A Business-tier heartbeat plan is $20 per month, or $240 a year ($20 times 12). The ITIC survey's floor for most mid-size and large firms is $300,000 for a single hour of downtime. Dividing, $300,000 by $240 is 1,250, so one hour at that floor pays for 1,250 years of the independent detector. Two caveats keep this honest: the survey is self-reported and measures downtime of the product, not the marginal cost of detecting it late, and the detector only helps if its alert reaches a person who can act. The cost of the detector is not the money; it is the routing, the on-call ownership and the quarterly drill.
Three things nobody has published. First, how often the external heartbeat receivers themselves fail or are misconfigured; the one data point is alerta's UI hiding expired heartbeats [18]. Second, the false-positive rate of Watchdog-absence pages, which determines whether anyone trusts them. Third, any measured mean-time-to-detect for a telemetry-plane outage with and without an external detector. The Microsoft study [34] establishes that detection gaps matter at hyperscale, but I did not take a figure from it.
Prices come from vendor pages that carry no date; the as-of date is the day I fetched them. The Statuspage figures came from a summarised fetch and are not verbatim. Durations marked derived are my subtraction from the timestamps each post gives, and use the post's own time zone. The ITIC figure is a self-reported survey band.
Every source behind this page, graded. Filter by kind. Gaps are stated rather than hidden: two talk timestamps could not be verified, several vendor pages are undated, and the Firebase account is secondary.
The backbone withdrawal made DNS servers unreachable, and the loss of DNS broke the internal tools used to investigate; primary and out-of-band network access were both down.
Internal network congestion on 7 December 2021 degraded real-time monitoring data for operators, delayed CloudWatch and impaired the Service Health Dashboard, which could not fail over to its standby region as intended.
Kinesis failure took CloudWatch metrics and alarms into INSUFFICIENT_DATA, and the Service Health Dashboard could not be updated because its posting tool used Cognito, which depended on Kinesis; a backup tool with fewer dependencies was used instead.
A 73-hour outage during which telemetry relied on the failing Consul, a circular dependency the company says severely hampered triage.
An external monitoring service paged first; then the dashboarding and alerting service became unavailable during triage, hampering investigation.
A data-centre power failure removed the logging and analytics cluster that was deliberately outside the HA cluster; observability tools did not detect that the power source had changed.
A systemd update applied automatically deleted CNI routes across five regions; at the start of the outage customer monitors were unavailable and not alerting.
Engineers were alerted two minutes after the failure began, but debugging was hampered by tools competing for the congested network.
Backup jobs failed silently on a version mismatch, and failure notification emails were rejected for missing DMARC, leaving no indication of failure.
The status page, hosted off Cloudflare, went down at the same time as the incident; the team initially read the pattern as an attack. Debugging and observability systems consumed large amounts of CPU.
Secondary account quoting Google's postmortem: the Firebase and Google Ads status dashboards stayed green because they relied on server-side health metrics and did not register client-side SDK crashes.
The status page resolved from the same DNS provider as python.org; the team committed to run DNS from a different provider for incidents like this.
The shipped Watchdog rule has expression vector(1), severity none, and a description that sends the reader to an external notification mechanism for the absence of the alert.
A proposal to change what the Watchdog expression measures was opened and closed by its author the same day; a maintainer asked for the expression to be configurable in the chart and mirrored in the hack scripts. No maintainer reason for closing is on the page.
A PR to let a single alert carry its own labels, motivated by labelling Watchdog with heartbeat: true for routing, was closed by its author on 2023-12-19 with all seven code-owner reviews still awaiting.
A user wanting a dead man's switch asked Alertmanager to send periodic heartbeats; a maintainer said switching the webhook to GET does not make sense.
The operator needed explicit support to ship the Watchdog to an OpsGenie heartbeat; the issue names the feature as 'also known as Watchdog/DeadManSwitch in other platforms'.
The receiving side had its own gap: the web UI did not indicate expired heartbeats.
The project's guidance opens its metamonitoring section by asking for confidence that monitoring works, and recommends external blackbox monitoring as a fallback (retrieved paraphrase plus fragments).
Grafana's guidance for Loki: collect data about Loki in a separate instance of Loki, Prometheus and Grafana, so a broken cluster can be diagnosed from a working one.
Names the practice meta-monitoring; the page does not say to use a separate cluster (so it is weaker than the Loki page).
The runbook says the Watchdog alert exists to ensure the entire alerting pipeline is functional and recommends an external integration that notifies when it stops firing, naming DeadMansSnitch in PagerDuty.
Page on symptoms, and ask that the elements of the monitoring system that direct to a pager be very simple and robust.
A team that decided to monitor its monitoring built a Terraform module for each step of the flow it wanted to observe.
Vendor blog stating that a stopped Prometheus cannot evaluate its own rules and that the prober must sit outside the failure domain.
Practitioner post: a self-check is only as trustworthy as the system running it.
A small-scale anecdote: a scheduled CI job updating the status page died for four days with no runner and no logs; a frozen status page does not look broken.
News write-up of Nanit's switch from a watcher-of-watchers to a vigilance control; the old setup contained a single point of failure.
Vendor advice: never host the status page within your own infrastructure; if self-hosting, use separate infrastructure, data centre, hosting provider and domain. The claim about Facebook's setup is the vendor's inference.
Weak, vendor-flavoured: verify data volume (events collected over the last two hours) and not only that the process is up.
Monarch trades consistency for availability because blocking on Spanner would raise mean-time-to-detection; it has an internal, an external and a 'meta' deployment, the meta running a proven-stable older version that monitors all other Monarch deployments.
Facebook's telemetry store sets availability above efficiency: two redundant in-memory copies and survival of whole-region loss, at the price of possibly dropping small amounts of data on the write path.
An out-of-band control network is not immune: its failures have significant impact; monitoring coverage gaps and collector placement hamper visibility.
A study of a year of Microsoft incidents on monitoring gaps and detection delay; generic evidence, no figure taken here.
Alerts that depend on a metric go silent when the metric disappears; the talk generates 'absent' alerts for metrics expected to exist. Slide text: 'We found metrics that weren't actually being exported'. Video timestamp not verified; claims rest on the session abstract and slide text.
A vendor employee argues for two independent monitoring systems on different providers; the transcript has no timestamps, so none is cited.
Heartbeat monitoring price points: Hobbyist $0 per month (20 jobs), Business $20 (100 jobs), Business Plus $80 (1,000 jobs).
Add-ons: 10 monitors and 10 heartbeats and 1 status page included; 10 more heartbeats $20 per month; an additional status page $15 per month.
Public-page tiers per month: Free, Hobby $29, Startup $99, Business $399, Enterprise $1,499 (from a summarised fetch, not verbatim).
A self-reported survey of over 1,000 firms: downtime exceeds $300,000 per hour for over 90% of mid-size and large enterprises.
Six rungs. The line between toy and real is rung four, where the receiver moves out of your laptop.
Compose file with Prometheus, Alertmanager and the kube-prometheus Watchdog rule (vector(1), severity none), routed to a local webhook that logs each delivery.
Done when: the webhook log shows a delivery at the repeat interval. Teaches: what the alert pipeline emits when everything is fine.
Stop Prometheus, then Alertmanager, then the receiver, one at a time. Write down which stops produce a page and which produce silence.
Done when: you have a table of stage, symptom, and whether a human was told. Teaches: which failures are loud and which are silent, in your own stack.
Point the webhook at an external heartbeat service (Healthchecks has a free tier) with a grace period of three repeat intervals. Repeat rung two.
Done when: stopping any stage produces a page from outside your laptop. Teaches: how silence becomes a signal, and what the grace period costs in detection time.
For every alert, add a companion using absent() or a freshness check on the exported metric, and initialise counters at start-up.
Done when: removing an exporter produces a named alert, not a quiet dashboard. Teaches: the Stile lesson: an alert is only as alive as its input.
List what the on-call needs from alert to fix: the pager, VPN or SSO, DNS, chat, the runbook host, the shell. Mark every shared dependency with the product.
Done when: each shared dependency has a documented alternative or an accepted risk. Teaches: the Facebook and Google lesson: detection and diagnosis are different systems.
Partition the telemetry plane from the product in a staging environment and run an incident with the monitoring dark. Time to first correct hypothesis is the metric. Repeat after every platform change.
Done when: the drill is on the calendar and its result changed at least one dependency. Teaches: that independence decays unless it is tested.
Two searches were run as written; the rest are built from vocabulary the sources taught and were used by the research agents or are suggested next steps. The page marks which is which.
Alertmanager "dead man's switch" OR "DeadMansSwitch" issue watchdog heartbeat alert pipeline github prometheuspostmortem monitoring system down during incident "we were blind" OR "flying blind" observability dependent on same infrastructure"dead man's switch" OR watchdog OR heartbeat OR "vigilance control" alerting pipeline"meta-monitoring" OR "metamonitoring" OR "monitoring the monitoring""circular dependency" telemetry postmortem"status page" "same DNS provider" OR "hosted on" postmortemrepo:prometheus-community/helm-charts is:pr is:closed is:unmerged watchdogintitle:"who watches the watchers" monitoring