Evidence ledger
40 sources
Checked 18 Oct 2026
Evidence ledger
One row per claim in When the monitor shares the outage: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Dig: When the monitor shares the outage: telemetry, alerting and status pages that fail with the system they watch. Research date: 2026-10-18. All rows checked 2026-10-18.
Method and limits, stated up front. Primary pages were fetched with WebFetch, which returns a model-written extract of the page rather than raw HTML. Quotes below are the extract's quotations; where the tool paraphrased, the row says "(retrieved summary)". Two research subagents fetched rows 10 to 40 under the same constraint and reported quotes verbatim from their fetches; rows 13 to 18 (GitHub) record the state shown on the page. Monarch (row 31) was read by extracting the downloaded PDF text. Not verified: talk video timestamps (rows 35, 36), the Facebook outage duration, the Google Firebase postmortem itself (row 11 is a newsletter quoting it), and any Cloudflare 2025 remediation detail. Pricing pages are undated; "published" for them is the fetch date.
Grading notes. Rows 19 to 22 are documentation, not decision records, and are graded adr because they state a
recommended design and a reason. Row 23 (the SRE book) is graded blog. Vendor-authored advice (25, 29, 30, 36 content)
is graded vendor or noted as vendor-employee. The classification into shared fate, silent pipeline and
blind-by-design is this guide's, not the sources'.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Meta (Facebook) | More details about the October 4 outage | postmortem | 2021-10-05 | 2026-10-18 | https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/ | Total loss of DNS broke the internal investigation tools; out-of-band access was also down | "the total loss of DNS broke many of the internal tools we'd normally use to investigate and resolve outages like this."; "Our primary and out-of-band network access was down" |
| 2 | AWS | Summary of the AWS Service Event in the N. Virginia (US-EAST-1) Region | postmortem | 2021-12-10 | 2026-10-18 | https://aws.amazon.com/message/12721/ | Congestion removed operators' monitoring visibility and impaired the status dashboard | "This congestion immediately impacted the availability of real-time monitoring data for our internal operations teams"; "the networking congestion impaired our Service Health Dashboard tooling" |
| 3 | AWS | Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region | postmortem | 2020-11-25 | 2026-10-18 | https://aws.amazon.com/message/11201/ | Status tooling depended on a failing service; fallback tool had fewer dependencies but was slower and less familiar | "we were unable to update the Service Health Dashboard because the tool we use to post these updates itself uses Cognito"; "alarms transitioned to the INSUFFICIENT_DATA state." |
| 4 | Roblox | Return to Service 10/28-10/31 2021 | postmortem | 2022-01 (event 2021-10-28 to 10-31) | 2026-10-18 | https://about.roblox.com/newsroom/2022/01/roblox-return-to-service-10-28-10-31-2021 | Circular dependency between telemetry and Consul; outage 73 hours | "There was a circular dependency between our telemetry systems and Consul"; "The outage lasted 73 hours." |
| 5 | Slack | Slack's Outage on January 4th 2021 | postmortem | 2021-01 (event 2021-01-04) | 2026-10-18 | https://slack.engineering/slacks-outage-on-january-4th-2021/ | Detected by an external monitor; internal dashboards and alerting lost mid-incident | "we got paged by an external monitoring service"; "our dashboarding and alerting service became unavailable."; "all debugging and investigation was now hampered by the lack of our usual dashboards and alerts." |
| 6 | Cloudflare | Post mortem on Cloudflare Control Plane and Analytics Outage | postmortem | 2023-11 (event 2023-11-02 to 11-04) | 2026-10-18 | https://blog.cloudflare.com/post-mortem-on-cloudflare-control-plane-and-analytics-outage/ | Logging excluded from HA cluster; observability could not see the power change | "None of our observability tools were able to detect that the source of power had changed."; "our logging systems were intentionally not part of the high availability cluster" |
| 7 | Datadog | Multiregion infrastructure connectivity issue | postmortem | 2023-03 (event 2023-03-08 to 03-10) | 2026-10-18 | https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ | Vendor outage across five regions; monitors unavailable and not alerting | "monitors were unavailable and not alerting"; "Starting on March 8, 2023, at 06:03 UTC, we experienced an outage" |
| 8 | Google Cloud | Google Cloud Networking Incident #19009 | postmortem | 2019-06 (event 2019-06-02) | 2026-10-18 | https://status.cloud.google.com/incident/cloud-networking/19009 | Alerted at 2 minutes; diagnosis slowed by tooling on the congested network | "Google engineers were alerted to the failure two minutes after it began"; "Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network." |
| 9 | GitLab | Postmortem of database outage of January 31 | postmortem | 2017-02 (event 2017-01-31) | 2026-10-18 | https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/ | Failure notifications were rejected; the sender had no other way to report | "Notifications were sent upon failure, but because of the Emails being rejected there was no indication of failure."; "This means we were never aware of the backups failing, until it was too late." |
| 10 | Cloudflare | Cloudflare outage on November 18, 2025 | postmortem | 2025-11-18 | 2026-10-18 | https://blog.cloudflare.com/18-november-2025-outage/ | Independent status page failed coincidentally; observability tooling consumed CPU | "Cloudflare's status page went down."; "large amounts of CPU being consumed by our debugging and observability systems" |
| 11 | The Pragmatic Engineer (quoting Google) | The Pulse: Firebase's global outage & poor response | blog | 2026-10-08 | 2026-10-18 | https://blog.pragmaticengineer.com/the-pulse-firebases-global-outage-poor-response/ | Dashboards green throughout a client-side SDK failure | "Throughout the outage, both the Firebase and Google Ads status dashboards remained green."; "Because these dashboards rely primarily on server-side health metrics, they did not register client-side SDK crashes." |
| 12 | Python Software Foundation | Infrastructure postmortem: DNS outage | postmortem | 2016-10-21 | 2026-10-18 | https://status.python.org/incidents/x97mmj5rqs5f | Status page resolved from the same DNS provider as the site | "resolved from the same DNS provider as the remainder of python.org."; "will run its DNS from a different provider for incidents like this in the future." |
| 13 | prometheus-operator/kube-prometheus | kubePrometheus-prometheusRule.yaml: the Watchdog alert | source | main branch (undated) | 2026-10-18 | https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/kubePrometheus-prometheusRule.yaml | Watchdog is vector(1), always firing, with an external detector for its absence | "This is an alert meant to ensure that the entire alerting pipeline is functional."; "This alert is always firing, therefore it should always be firing in Alertmanager" |
| 14 | prometheus-community/helm-charts | PR #4753 [kube-prometheus-stack] Improve watchdog alert (closed, not merged) | source | 2024-07-29 | 2026-10-18 | https://github.com/prometheus-community/helm-charts/pull/4753 | Rejected/closed-unmerged PR over what the Watchdog should prove | Maintainer comment (partial): wants the expression "configurable within the chart" and said it "needs to be part of the hack scripts (the python scripts in that hack folder), too." (retrieved summary of the PR) |
| 15 | prometheus-community/helm-charts | PR #4037 [kube-prometheus-stack] allow setting labels per Alert (closed, not merged) | source | 2023-11-23 | 2026-10-18 | https://github.com/prometheus-community/helm-charts/pull/4037 | Closed-unmerged PR needed to route Watchdog to a heartbeat receiver | "allows setting labels that should only be applied for a certain alert." (retrieved summary of the PR) |
| 16 | prometheus/alertmanager | Issue #2407 Feature request: support GET for webhook receiver (closed as completed 2020-11-12) | source | 2020-10-29 | 2026-10-18 | https://github.com/prometheus/alertmanager/issues/2407 | Alertmanager has no native heartbeat sender | "I wanted to create 'dead man's switch' alert for prometheus and alertmanager."; "I expected the heartbeat to be updated periodically by alertmanager." |
| 17 | prometheus-operator/prometheus-operator | Issue #3970 Support OpsGenie Heartbeat (closed 2021-09-06 via PR #4110) | source | 2021-04-09 | 2026-10-18 | https://github.com/prometheus-operator/prometheus-operator/issues/3970 | Heartbeat support was a feature request landed five months later | "also known as Watchdog/DeadManSwitch in other platforms"; "is an important feature to monitor if the Prometheus cluster is online." |
| 18 | alerta/alerta | Issue #1110 Improve Prometheus Heartbeat handling (DeadMansSwitch) (closed 2020-06-07) | source | 2019-11-07 | 2026-10-18 | https://github.com/alerta/alerta/issues/1110 | The heartbeat receiver UI did not surface expired heartbeats | "there is no indicator in the heartbeat WebUI view, that shows expired heartbeats" |
| 19 | Prometheus project | Alerting practices: Metamonitoring | adr | undated | 2026-10-18 | https://prometheus.io/docs/practices/alerting/ | External blackbox monitoring catches otherwise invisible problems | "It is important to have confidence that monitoring is working."; "external blackbox monitoring can catch problems that are otherwise invisible"; "serves as a fallback in case internal systems completely fail." |
| 20 | Grafana Labs (Loki docs) | Meta monitoring (Loki operations) | adr | undated | 2026-10-18 | https://grafana.com/docs/loki/latest/operations/meta-monitoring/ | Collect telemetry about the system in a separate instance | "As a best practice, you should collect data about Loki in a separate instance of Loki, Prometheus, and Grafana."; "This will let you troubleshoot a broken Loki cluster from a working one." |
| 21 | Grafana Labs (Mimir docs) | Monitor system health (Mimir, next docs) | adr | undated | 2026-10-18 | https://grafana.com/docs/mimir/next/manage/monitor-grafana-mimir/monitor-system-health/ | Meta-monitoring named; separation not required on the page | "This process is called meta-monitoring." |
| 22 | Prometheus Operator community | Runbook: Watchdog | adr | undated | 2026-10-18 | https://runbooks.prometheus-operator.dev/runbooks/general/watchdog/ | Mitigation is an external notifier on absence of the alert | "This is an alert meant to ensure that the entire alerting pipeline is functional."; page names the DeadMansSnitch integration in PagerDuty |
| 23 | Site Reliability Engineering, chapter 6: Monitoring Distributed Systems | blog | 2016 | 2026-10-18 | https://sre.google/sre-book/monitoring-distributed-systems/ | Pager-directing elements must be very simple and robust | "the elements of your monitoring system that direct to a pager need to be very simple and robust."; "Every page should be actionable." | |
| 24 | Mercari | Who Watches the Watchmen? Keeping an Eye on Our Monitoring Systems | blog | 2022-08-05 | 2026-10-18 | https://engineering.mercari.com/en/blog/entry/20220805-who-watches-the-watchmen-keeping-an-eye-on-our-monitoring-systems/ | Per-stage Terraform modules for monitoring the monitoring | "decided it was high time to begin monitoring our monitoring to prevent these kinds of issues occuring."; "We created Terraform modules for each of the steps in the flow we wanted to observe." |
| 25 | OneUptime | Monitor Prometheus So Its Failure Cannot Silence Host-Down Alerts | vendor | 2026-07-31 | 2026-10-18 | https://oneuptime.com/blog/post/2026-07-31-monitor-prometheus-server-out-of-band/view | A stopped Prometheus cannot evaluate its own alert rules | "A Prometheus server cannot evaluate its own alert rules while it is stopped."; "Place the prober and the Prometheus evaluating this expression outside the target's failure domain." |
| 26 | Irin Observability (DEV Community) | A Dead Man's Switch for Your Monitoring Stack | blog | undated (Jul 29) | 2026-10-18 | https://dev.to/irinobservability/a-dead-mans-switch-for-your-monitoring-stack-2335 | A self-check is only as trustworthy as its host | "A self-check is only ever as trustworthy as the system running it."; "the same outage that broke your alerting also broke the panel reporting on your alerting." |
| 27 | haruodev (DEV Community) | Our status page froze for four days because the CI minutes ran out | blog | undated (Sep 8) | 2026-10-18 | https://dev.to/haruodev/our-status-page-froze-for-four-days-because-the-ci-minutes-ran-out-1m37 | Monitor watched the wrapper; scheduled job died silently | "If production had gone down, nothing would have told anyone."; "A frozen status page does not look broken."; "The monitor was watching the wrapper." |
| 28 | InfoQ (Nanit) | Nanit replaces monitoring-of-monitoring with a Dead Man's Switch (InfoQ news) | blog | 2022-05-19 | 2026-10-18 | https://www.infoq.com/news/2022/05/monitoring-monitoring/ | Nanit used a vigilance control to monitor its monitoring | "We're using a vigilance control as a way to monitor our monitoring systems."; the old setup "contained a single point of failure in the design." |
| 29 | StatusPal | Learning from Facebook: Keep your Status Page Separately from your Infrastructure | vendor | 2021-04-09 | 2026-10-18 | https://statuspal.io/blog/2021-04-09-learning-from-facebook-keep-your-status-page-separately-from-your-infrastructure | Vendor advice to host the status page separately | "never host your status page within your own infrastructure." |
| 30 | Zabbix | Zabbix and SCADA: Who watches the watchers? | vendor | 2020-07-01 | 2026-10-18 | https://blog.zabbix.com/zabbix-and-scada-who-watches-the-watchers/11117/ | Check data-flow volume as well as liveness | "Zabbix monitors the database to verify how many events have been collected over the last two hours." |
| 31 | Google (Adams et al.) | Monarch: Google's Planet-Scale In-Memory Time Series Database (PVLDB 13(12)) | paper | 2020-08 | 2026-10-18 | https://www.vldb.org/pvldb/vol13/p3181-adams.pdf | A meta deployment on a proven-stable older version monitors all other deployments | "meta runs a proven-stable older version of Monarch and monitors all other Monarch deployments."; "Writing to or reading from a strongly consistent database like Spanner may block for a long time; that is unacceptable for Monarch because it would increase mean-time-to-detection and mean-time-to-mitigation for potential outages." |
| 32 | Facebook (Pelkonen et al.) | Gorilla: A Fast, Scalable, In-Memory Time Series Database (PVLDB 8(12)) | paper | 2015-09 | 2026-10-18 | https://www.vldb.org/pvldb/vol8/p1816-teller.pdf | Telemetry store: availability over efficiency, survives region loss | "High availability trumps resource efficiency."; "Our primary requirement for a TSDB is that it should always be available to take writes." |
| 33 | Google (Govindan et al.) | Evolve or Die: High-Availability Design Principles Drawn from Google's Network Infrastructure (SIGCOMM 2016) | paper | 2016-08 | 2026-10-18 | https://homepages.dcc.ufmg.br/~cunha/teaching/20221/cloudcomp/readings/03-readings/govindan16evolve.pdf | Out-of-band network failures matter; collector placement hampers visibility | "We use an out-of-band control plane network for the fabrics and B4, and failures in this network can have significant impact on the control planes and, therefore, on availability." |
| 34 | Microsoft (Parayil et al.) | Detection Is Better Than Cure: A Cloud Incidents Perspective (ESEC/FSE 2023) | paper | 2023-12 | 2026-10-18 | https://www.microsoft.com/en-us/research/?p=990273 | Improper monitoring delays detection and mitigation | "Improper monitoring can lead to delays in the detection and mitigation of production incidents." |
| 35 | Stile Education (Nick Spain) | Watching the Watchers: Generating Absent Alerts for Prometheus (SREcon21) | talk | 2021-10-13 | 2026-10-18 | https://www.usenix.org/conference/srecon21/presentation/spain | Alerts do not fire when the metrics go away | "Your Prometheus alerts aren't going to fire and you won't know that they've gone away." |
| 36 | VictoriaMetrics (Mathias Palmersheim) | How to Monitor your Monitoring (Conf42 Incident Management 2024) | talk | 2024-10-17 | 2026-10-18 | https://conf42.com/Incident_Management_2024_Mathias_Palmersheim_27_monitoring_monitoring_how | Run two independent monitors on different providers | "If your monitoring system falls over in the middle of the night, does somebody get paged at 2 AM?"; "different infrastructure providers don't have simultaneous outages." |
| 37 | Healthchecks.io | Pricing | vendor | 2026-10-18 (fetched; page undated) | 2026-10-18 | https://healthchecks.io/pricing/ | Business plan $20 per month for 100 jobs | "Business" "$20 / month" "Monitor 100 jobs" |
| 38 | Better Stack | Pricing | vendor | 2026-10-18 (fetched; page undated) | 2026-10-18 | https://betterstack.com/pricing | 10 heartbeats included; 10 more $20 per month | "10 heartbeats included"; additional 10 heartbeats "$20 per month" |
| 39 | Atlassian | Statuspage pricing | vendor | 2026-10-18 (fetched; page undated) | 2026-10-18 | https://www.atlassian.com/software/statuspage/pricing | Public page tiers Hobby $29 to Enterprise $1,499 per month | Retrieved summary of the pricing page; no verbatim quote captured |
| 40 | ITIC (via Calyptix) | 2024 Hourly Cost of Downtime Survey, Part 2 | vendor | 2024 | 2026-10-18 | https://calyptix.com/wp-content/uploads/ITIC-2024-Hourly-Cost-of-Downtime-Survey-Results-Part-2.pdf | Hourly downtime above $300,000 for over 90% of mid-size and large firms | "downtime now exceeds $300,000 for over 90% of mid-size and large enterprises." |