The update that expires: ten years of Datadog on machines it does not own
How Datadog changed the software it installs on customers' machines between 2016 and 2026: the compatibility shim that made the Go rewrite survivable, the signed remote change channel, the remote upgrade that expires by default, the per-host rehearsal, and the install-base tail that certificate rotation keeps alive.
A decade of one observability company read through the only part of it that runs on other people's computers, reconstructed from commit history, systemd unit templates, CI budget files, closed pull requests and the single outage Datadog published a root cause for. An architect leaves able to name the four mechanisms that make a change to unreachable software safe, and to price the tail every one of them leaves behind.
The commit rate on the abandoned Agent 5 repository went back up after 2022, and the work is certificate-authority rotation runbooks written for bash 3.1 on RHEL 5 and PowerShell 2.0 on Windows Server 2008 R2: an install-base tail is kept alive by expiring cryptography, not by demand for features.
What you get out of it
- The control channel came first: Datadog could change Agent behaviour remotely from December 2021, three years before it could change the binary in April 2024, and two years before it could rehearse the change on the target host in August 2026.
- A remotely started Agent version expires. The experiment systemd unit runs under a 3000 second timeout, declares OnFailure back to the stable unit and forces a failed state on stop, so doing nothing reverts the host.
- The March 2023 outage came through a channel Datadog did not operate: an automatic OS security update whose systemd-networkd restart deleted Cilium's routes on tens of thousands of nodes across three cloud providers. The same mechanism had been filed upstream four months earlier and closed as not-a-bug, and the genuine defect was not fixed until March 2025.
- The footprint promised to a customer is enforced as a build failure: 179 MiB of total memory for an idle install, 537 MiB with every feature enabled, 355 MiB and 100 millicores for the security agent.
- Every mechanism assumes the customer may decline it, and on at least one configuration declining is broken: with remote configuration disabled, the installer service crash-loops with exit code 255 (issue #43052, open since November 2025).
Scope
Why this, now. Fleet-resident software is spreading well beyond monitoring agents, into security sensors, AI runtimes and auto-instrumentation libraries, and the mechanisms that make remote change survivable are now visible in a public repository rather than described in a vendor deck.
What it does not cover. Datadog's backend beyond the two figures needed to size the ingest problem, feature-by-feature comparison with competing agents, and any assessment of Datadog as a business; comparable vendors' engineering accounts were not reachable from this session, so cross-vendor generalisation is marked as inference.
Other field guides
Where Shopify refused to rewrite, and where it rewrote three times
One company running two opposite policies in the same decade. On the substrate it can change, Shopify has not migrated since 2014: it funded a JIT in…
32 sources · 9 organisations · 4 postmortemsChanging a platform without stopping it: a decade of Zalando's cluster fleet
Zalando's production cluster configuration has been a public repository since 2017, and it took a change on the day this guide was written. The guide…
20 sources · 2 organisations · 2 postmortemsThe half-life of in-house infrastructure: a decade of Uber, read from its own repositories
A decade of one company's platform decisions read from the artefacts nobody edits on purpose: archive banners, last-commit dates, closed-unmerged pul…
26 sources · 8 organisations · 4 postmortems