Evidence ledger
28 sources
Checked 12 Sep 2026
Evidence ledger
One row per claim in The update that expires: ten years of Datadog on machines it does not own: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Field guide: The update that expires: ten years of Datadog on machines it does not own Research date: 2026-09-12. All links fetched on 2026-09-12.
Method note: this session's network egress policy reaches code hosts (github.com, raw.githubusercontent.com), Datadog's own blog and documentation, and its status page. Most third-party engineering blogs, arxiv.org, usenix.org and web.archive.org were blocked, so the corpus is deliberately weighted toward artefacts nobody edits for effect: commit history, release tags, systemd unit templates, CI budget files, issue threads and pull requests. Where a claim rests on a single Datadog-authored account, the ledger says so.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Datadog | 2023-03-08 incident: root cause | postmortem | 2023-05-16 | 2026-09-12 | https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ | An unscheduled OS security update, not a Datadog change, took out five regions across three clouds | "on March 8, 2023, at 06:00 UTC, a security update to systemd was automatically applied to a number of VMs, which caused a latent adverse interaction in the network stack" |
| 2 | Datadog | 2023-03-08 incident: root cause | postmortem | 2023-05-16 | 2026-09-12 | https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ | The mechanism was systemd-networkd deleting routes owned by the CNI plugin | "systemd-networkd forcibly deleted the routes managed by the Container Network Interface (CNI) plugin (Cilium) we use for communication between containers." |
| 3 | Datadog | 2023-03-08 incident: root cause | postmortem | 2023-05-16 | 2026-09-12 | https://www.datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue/ | Blast radius and the remediation that names the channel itself | "This affected tens of thousands of nodes in our fleet between 06:00 and 07:00"; remediation "disabled this legacy security update channel across all affected regions" and "audited our infrastructure for potential similar legacy update channels" |
| 4 | Datadog | 2023-03-08 incident: a deep dive into our incident response | postmortem | 2023-06-01 | 2026-09-12 | https://www.datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-incident-response/ | Recovery cost measured in people and days, not minutes | "Within the first hour, we had more than 50 engineers involved in the response."; "We would eventually open almost 100 workstreams, which involved between 500 and 750 engineers working in shifts" |
| 5 | Datadog | Hardening eBPF for runtime security: Lessons from Datadog Workload Protection | postmortem | 2026-01-07 | 2026-09-12 | https://www.datadoghq.com/blog/engineering/ebpf-workload-protection-lessons/ | Datadog's own kernel-resident code collided with the host's CNI and broke pod connectivity | A race condition caused Datadog's TC filters to disrupt Cilium's network filters, "breaking pod connectivity" until manual restart |
| 6 | Datadog | Hardening eBPF for runtime security | postmortem | 2026-01-07 | 2026-09-12 | https://www.datadoghq.com/blog/engineering/ebpf-workload-protection-lessons/ | Guardrails for code loaded into other people's kernels | agent "refuses to start and provides a clear, actionable error" if critical eBPF programs fail to load; "runs self-tests at startup and reports results to our backend"; supports "all kernels down to version 4.15"; drops "up to 95% of captured events" in kernel |
| 7 | Datadog | How we reduced the size of our Agent Go binaries by up to 77% | blog | 2026-02-18 | 2026-09-12 | https://www.datadoghq.com/blog/engineering/agent-go-binaries/ | Binary size is a customer-visible cost when the software lands on millions of hosts | Core Agent "236 MiB → 103 MiB (decrease of 56%)"; package "from 1.22 GiB to 688 MiB uncompressed"; "we maintain dozens of builds that vary based on operating system (OS), architecture, and specific distribution targets" |
| 8 | Datadog | How we reduced the size of our Agent Go binaries | blog | 2026-02-18 | 2026-09-12 | https://www.datadoghq.com/blog/engineering/agent-go-binaries/ | Why they cared | "This growth impacted both us and our users: network costs and resource usage increased, perception of the Agent worsened, and it became harder to use the Agent on resource-constrained platforms." |
| 9 | Datadog | Introducing Husky, Datadog's third-generation event store | blog | 2022-05-17 | 2026-09-12 | https://www.datadoghq.com/blog/engineering/introducing-husky/ | The server side moved to object storage to escape per-tenant noisy-neighbour effects | "A single tenant bursting and emitting a huge number of events in a short period of time could lead to query degradation for all other tenants colocated on the same shard."; migration took "over a year and a half" |
| 10 | Datadog | Inside Husky's query engine: real-time access to 100 trillion events | blog | 2025-10-01 | 2026-09-12 | https://www.datadoghq.com/blog/engineering/husky-query-architecture/ | Backend scale as of late 2025 | "more than 100 trillion events and billions of queries every day"; result cache hit ratio "80%"; "3.4% of queries end up scanning real data" |
| 11 | Datadog | Agent Version differences (docs) | vendor | undated page | 2026-09-12 | https://docs.datadoghq.com/agent/guide/version_differences/ | The vendor's stated upgrade expectation, against which the installed tail is measured | "Datadog recommends you update Datadog Agent with every minor and patch release, or, at a minimum, monthly."; "Agent 6 is a complete rewrite of the core Agent in Golang." |
| 12 | Datadog | Fleet Automation (docs) | vendor | undated page | 2026-09-12 | https://docs.datadoghq.com/agent/fleet_automation/ | Remote management of installed agents is now a product surface | "Datadog Fleet Automation allows you to centrally govern and remotely manage Datadog Agents and OpenTelemetry (OTel) Collectors at scale"; "Keep your fleet up to date by identifying and upgrading outdated Agent and OTel Collector versions." |
| 13 | Datadog | dd-agent README (Agent 5 repository) | source | last touched 2026-07-01 | 2026-09-12 | https://raw.githubusercontent.com/DataDog/dd-agent/master/README.md | The Python 2 agent line is still declared supported in 2026 | "This repository contains the source code for the Datadog Agent up to and including major version 5. Although still supported, no major feature is planned for this release line"; setup requires "python 2.7" |
| 14 | Datadog | dd-agent CHANGELOG | source | 5.32.9 released 2023-01-12 | 2026-09-12 | https://raw.githubusercontent.com/DataDog/dd-agent/master/CHANGELOG.md | The last release of the old line existed only to re-sign a package | "# 5.32.9 / 2023-01-12 ... This release is specific to the RHEL/CentOS RPM package, and is signed with the current RPM signing key ... No other changes are included" |
| 15 | Datadog | Agent v5 certificate update runbook | source | 2026-03-10 (commit) | 2026-09-12 | https://raw.githubusercontent.com/DataDog/dd-agent/master/runbooks/sectigo-root-ca-rotation-2025/README.md | Expiring cryptography, not features, is what forces work on the frozen tail | "Agent v5 uses an embedded certificate bundle for SSL/TLS verification. When Datadog's SSL certificates are updated to use newer certificate authorities, older Agent v5 installations may not recognize these certificates, causing the Agent to lose connectivity with Datadog." |
| 16 | Datadog | dd-agent commit: legacy OS scripts for sectigo CA rotation | source | 2026-03-10 | 2026-09-12 | https://github.com/DataDog/dd-agent/commit/2330e07b2910c3ff24b4a0aad610f970d00ce8e2 | In 2026 the oldest supported hosts are RHEL ⅚ and Windows Server 2008 R2 | "el5-6/linux.sh: supports RHEL/CentOS/Oracle Linux 5 and 6 - bash 3.1 compatible"; "windows-2008-2012/windows.ps1: supports Windows Server 2008 R2 and 2012 R2 - PS 2.0+ compatible" |
| 17 | Datadog | rtloader README | source | current main | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/rtloader/README.md | The Go rewrite kept an embedded CPython, switchable at runtime, to preserve the check ecosystem | "RtLoader is a C++ wrapper around the CPython API with a C89-compatible public API that can be used by foreign languages like Go ... Which Python version to use can be decided at runtime, RtLoader will dlopen the proper backend libraries accordingly."; "Python2 isn't supported anymore." |
| 18 | Datadog | Commit: rtloader: drop python2 support (#30089) | source | 2024-10-15 | 2026-09-12 | https://github.com/DataDog/datadog-agent/commit/4b8b2b5ab90d0bbc4732d4dca5b6aaabcd24159a | Dated end of Python 2 inside the agent, four years after upstream Python 2 EOL | Commit subject "rtloader: drop python2 support (#30089)", authored 2024-10-15 |
| 19 | Datadog | datadog-agent systemd unit template | source | current main | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/pkg/fleet/installer/packages/embedded/tmpl/datadog-agent.service.tmpl | A remotely started agent version expires by default and reverts to stable on failure | Experiment branch sets Environment="EXPERIMENT_TIMEOUT=3000s", OnFailure=datadog-agent.service, ExecStart=/usr/bin/timeout --kill-after=15s $EXPERIMENT_TIMEOUT ..., ExecStopPost=/bin/false; the stable branch has none of these |
| 20 | Datadog | Installer package hooks README | adr | current main | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/pkg/fleet/installer/packages/README.md | The three-state upgrade protocol: start experiment, stop, promote | "The installer supports a safer upgrade path for oci packages called \"experiments\"."; "Start experiment ... v2's files are written to disk. v1's files are kept intact."; "Promote experiment ... v1's files are removed from disk." |
| 21 | Datadog | Uptane client (remote configuration) | source | first commit 2021-12-17 | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/pkg/config/remote/uptane/client.go | The remote change channel is verified with the automotive OTA framework, not plain TLS | "Package uptane contains the logic needed to perform the Uptane verification checks against stored TUF metadata and the associated config files." |
| 22 | Datadog | Installer default package list | source | current main | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/pkg/fleet/installer/default_packages.go | Remote install now covers tracing libraries in eight languages, and carries version pins for a distribution that reached EOL in 2020 | {Name: "datadog-agent", ... releasedWithRemoteUpdates: true}; "Default versions pinned for CentOS 6 ... No manual testing is done on CentOS 6, so we can't guarantee that the latest versions will continue working." |
| 23 | Datadog | Quality gate: idle agent | source | gates named 2024-10-18 | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/test/regression/cases/quality_gate_idle/experiment.yaml | The agent's footprint on a customer machine is a hard CI bound, not a goal | "memory_allotment: 233 MiB"; check memory_usage "upper_bound: \"179 MiB\""; "Memory usage quality gate. This puts a bound on the total agent memory usage." |
| 24 | Datadog | Agent Data Plane preflight mode design note | adr | 2026-08-11 (commit) | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/docs/dev/agent-data-plane.md | The 2026 answer to untestable hosts: rehearse the change on the host and report the result as telemetry | "Preflight mode answers a single question before ADP is enabled for real on a given host: would it have started cleanly here? Historically the only way to find out was to set data_plane.enabled: true on a customer, which is the worst possible time to discover an OS, permissions, proxy or packaging problem." |
| 25 | Datadog | OpenTelemetry converter component README | adr | current main | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/comp/otelcol/converter/README.md | Datadog rewrites the customer's own collector configuration, with a documented opt-out | "The converter enhances the user provided configuration."; features list including infraattributes, ddflare, and "Disable all features" example |
| 26 | Datadog | PowerShell check allowlist example | source | current main | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/cmd/agent/dist/protected/powershell_allowlist.yaml.example | When the agent executes host commands, the host's administrator owns the policy and the default is to collect nothing | "The allowlist must be owned by Administrators or SYSTEM, or the check refuses to load it and collects nothing (fail closed)."; "no cmdlet is permitted until you create it" |
| 27 | Datadog | Agent CHANGELOG (6.0.0 to 7.83.1) | source | 7.83.1 on 2026-09-09 | 2026-09-12 | https://raw.githubusercontent.com/DataDog/datadog-agent/main/CHANGELOG.rst | The released record of what breaks on customer machines | Counted across the file on 2026-09-12: 76 occurrences of "panic", 61 of "crash", 17 of "memory leak", 16 of "deadlock" |
| 28 | Datadog | PR #29560: Add FIPS support to the Agent (closed unmerged) | source | opened 2024-09-25, closed 2024-11-07 | 2026-09-12 | https://github.com/DataDog/datadog-agent/pull/29560 | A change was rejected because it assumed host conditions the agent cannot assume | Review objection that running FIPS self-tests in the entrypoint script "would fail on read-only filesystems and when users lacked root privileges"; a "6.22% memory increase in the file_tree experiment" flagged by the regression detector; closed "in favor of this branch" |
| 29 | Datadog | Issue #43052: Failed to start datadog-agent-installer.service | source | opened 2025-11-13, open | 2026-09-12 | https://github.com/DataDog/datadog-agent/issues/43052 | Turning the remote channel off leaves a crash-looping service: the opt-out path is not exercised | Service fails with "Error: remote config is required to create the updater", exit code 255, on Agent 7.72.2 with remote_configuration: enabled: false |
| 30 | Datadog | PR #50607: forward Agent error logs to internal telemetry | source | 2026-06-19 | 2026-09-12 | https://github.com/DataDog/datadog-agent/pull/50607 | How a vendor observes a fleet it does not own without shipping customer data | "Adds a new experimental Agent ERROR logs forwarding to internal telemetry, so Datadog Engineers can monitor Agent error patterns at scale."; "Off by default, gated by: errortracking: enabled: true"; message dropped "to avoid leaking sensitive data" |
| 31 | systemd | Issue #25451: systemd deleting custom routes post restart | source | 2022-11-20, closed | 2026-09-12 | https://github.com/systemd/systemd/issues/25451 | The mechanism behind Datadog's 2023 outage was reported upstream four months earlier and closed as not-a-bug | Title "Systemd deleting custom routes set via ip route add post restart of systemd-networkd"; labels "network, not-a-bug" |
| 32 | systemd | PR #36756: fix unexpected removal of routes on restart even when KeepConfiguration=yes | source | merged 2025-03-15 | 2026-09-12 | https://github.com/systemd/systemd/pull/36756 | The genuine defect in the same area was fixed two years after the outage | Title "network: fix unexpected removal of routes on restart even when KeepConfiguration=yes", merged into systemd:main 2025-03-15 |
| 33 | Bottlerocket (AWS) | Issue #3436: CNI rules and routes lost when systemd-networkd restarts | source | 2023-09-08, closed | 2026-09-12 | https://github.com/bottlerocket-os/bottlerocket/issues/3436 | A second organisation hit the same class independently | After systemctl restart systemd-networkd, "all custom IP rules are removed, leaving only the default routing rules" |
| 34 | k3s | Issue #7736: install script breaks networking when using cilium with systemd >= 249 | source | 2023-06-11, closed | 2026-09-12 | https://github.com/k3s-io/k3s/issues/7736 | A third organisation, same class: two components managing the same kernel state | "install script breaks networking when using cilium with systemd >= 249"; workaround is INSTALL_K3S_SKIP_START |
| 35 | Datadog | integrations-core repository | source | first commit 2015-12-01 | 2026-09-12 | https://github.com/DataDog/integrations-core | The compatibility surface only grows | Counted from the repository tree on 2026-09-12: directories containing manifest.json numbered 102 at 2018-09-01, 151 at 2020-09-01, 196 at 2022-09-01, 260 at 2024-09-01 and 401 at 2026-09-01; 19 manifest deletions in the whole history |
| 36 | Datadog | datadog-agent repository | source | 6.0.0 tagged 2018-02-26 | 2026-09-12 | https://github.com/DataDog/datadog-agent | Release cadence and the length of the dual-line overlap | Counted from tags on 2026-09-12: 130 X.Y.0 tags; 6.0.0 on 2018-02-26; first 7.x (7.16.0) on 2019-12-17; last 6.x minor (6.61.0) on 2025-01-10; 62 minor releases on the 6 line |
Tier mix
postmortem 6 rows (3 distinct documents), source 22, adr 3, blog 4, vendor 2, paper 0, talk 0, casestudy 0.
Gaps in this corpus, stated plainly
- No papers and no talks. Datadog Research publications, conference recordings and arxiv were unreachable from this session, so nothing in the guide rests on one.
- One incident, two documents. Datadog has published a full root-cause account for exactly one outage (March 2023). Everything else about how its own fleet fails is inferred from release notes, issue threads and the status page.
- Single-vendor view. Comparable agents (New Relic, Elastic, Splunk, the upstream OpenTelemetry Collector) were not reachable; where the guide claims a pattern is general, it says so as inference rather than as a survey.