The update that expires: ten years of Datadog on machines it does not own

The territory

Software that runs inside someone else's operating system, on hardware you cannot log into, updated only with their consent. Everything in this guide follows from those three constraints.

3000s

Lifetime of a remotely started Agent version before the host kills it and restarts the old one

systemd unit template, main branch, 2026

179 MiB

CI upper bound on total memory for an idle Agent install

quality_gate_idle, 2026

tens of thousands

Nodes lost in one hour when an OS security update nobody scheduled removed container routes

Datadog root cause, May 2023

401

Third-party integrations the Agent must keep working, up from 102 in 2018

integrations-core tree, counted 2026-09-12

The interesting thing about Datadog's decade is not the backend, which is a well-told story of object storage and columnar formats, and not the product surface, which grew the way every platform company's does. It is that Datadog's business depends on a piece of software it writes but does not operate: a process installed on every host, container, function and laptop belonging to its customers. Datadog cannot log into those machines. It cannot force an upgrade. Until 2024 it could not perform one at all. And it cannot take a release back once the package is on disk.

That constraint produces an architecture you can read directly out of the repository, and the reading is unusually clean because the artefacts are not marketing. A systemd unit template states what happens when a new version fails to start. A CI file states the exact number of mebibytes the Agent may occupy on an idle machine. A commit dated March 2026 states which operating systems still have to be supported, because it contains a shell script written to run under bash 3.1.

The finding that reframed the rest

The repository for Agent 5, the Python 2 line superseded in 2018, has a commit rate that went back up. Two commits in 2022, six in 2023, nine in 2024, fourteen in 2025, twelve in the first nine months of 2026. None of them are features. They are a re-signed RPM in January 2023 and, in 2026, certificate-authority rotation runbooks with bash 3.1 compatibility for RHEL 5 and PowerShell 2.0 compatibility for Windows Server 2008 R2. The tail of an install base is not kept alive by demand for features. It is kept alive by cryptography that expires on someone else's schedule.

Read forward from that, and the decade has a shape. Datadog spent it converting the Agent from software the customer installs and updates into software Datadog changes, remotely, with the customer's standing consent, under mechanisms designed to make each change reversible without anyone logging in. The order matters: the ability to change behaviour arrived in December 2021, three years before the ability to change the binary in April 2024, and the ability to rehearse a change on the target host before making it arrived in August 2026, two years after that.

Scope. This guide covers software distributed to and updated on machines the vendor does not control, using Datadog's Agent as the worked example, plus the dependency collisions that follow from living inside someone else's OS. It does not cover Datadog's backend beyond the two figures needed to size the problem, does not compare Agent features with competing products, and does not assess Datadog as an investment or an employer. Where it claims a pattern generalises beyond this one company, it says so as inference, because the engineering blogs of comparable vendors were not reachable from this session.

Figure 1 · Ten years, read as the acquisition of channels

Each step is dated from the repository, not from a press release. Notice that the control channel precedes the update channel by three years, and that the oldest supported install base outlives both. Sources: datadog-agent tags and commits, dd-agent commit, March 2026.

Diagram source

How it is actually built

Seven parts recur in any software that lives on machines its author cannot reach. Six of them exist in the Datadog Agent today and can be pointed at in the repository; the seventh is the part every vendor gets wrong.

The shape below is reconstructed, not published. Datadog has no architecture decision record directory and no design-doc index. What it has is a package-hooks README that describes a three-state upgrade protocol, a set of generated systemd units that encode what happens when each state fails, a signed-metadata client borrowed from automotive over-the-air updates, and a CI directory that treats the customer's memory as a budget. Put those together and the architecture is legible.

Figure 2 · Reference architecture for software on machines you do not own

The boundary that matters is not the network, it is the machine: everything below the dashed line runs at the host owner's discretion and can be switched off by them. Reconstructed from the installer package hooks README, the Uptane client and the installer package list.

Diagram source

Two slots, one supervisor

Packages install into a versioned directory with a stable and an experiment path. Starting an experiment writes the new version to disk and leaves the old files intact; promoting deletes the old one; stopping deletes the new one. The state machine is in the package hooks, and the enforcement is in the unit files: the two services declare Conflicts on each other, so the host, not the vendor, guarantees that exactly one is running.

Read it at: packages/README.md

A change channel with its own trust model

Remote configuration is not an HTTPS poll with a JSON body. Since December 2021 the Agent verifies what it receives with Uptane, the framework designed for pushing software to cars, which separates the metadata that says what exists from the metadata that says what this particular client should run. A channel that can change behaviour on a million machines needs a threat model in which the vendor's own backend is one of the adversaries.

Read it at: uptane/client.go

A back channel that carries evidence, not logs

The vendor cannot see the host, so the host must describe itself. The Agent reports inventories, eBPF self-test results and, since August 2026, the outcome of a rehearsal run. A 2026 pull request adds forwarding of Agent error logs to Datadog's internal telemetry, off by default, with the message text dropped before send to avoid carrying customer data. The design tension is permanent: the vendor needs the failure detail and is not entitled to the contents.

Read it at: PR #50607

Three further parts are less visible and more load-bearing. The first is the compatibility layer. When Datadog rewrote the Agent core in Go for version 6, it did not rewrite the checks: it embedded CPython behind a C89 interface called rtloader, which picks a Python 2 or Python 3 backend with dlopen at runtime. That choice bought the migration five years of overlap and cost a C++ shim that lived until October 2024. The second is the resource budget, expressed as CI gates rather than as intent: 179 MiB of total memory for an idle install, 537 MiB with every feature on, 355 MiB and 100 millicores for the security agent. A pull request that exceeds them fails, which is how a 2024 attempt to add FIPS support was flagged for a 6.22 percent memory increase before any customer saw it. The third is the owner-controlled policy surface: when the Agent gained the ability to run PowerShell as a check, the allowlist was placed in a directory writable only by Administrators, and the check refuses to load a file that is not owned by Administrators or SYSTEM. The default is to collect nothing.

The seventh part is the one to design deliberately, because Datadog's own record shows what happens when it is left implicit: the inventory of every other channel that can change the host. In March 2023 an OS security update channel that Datadog's own fleet still had enabled, and that nobody was tracking, applied a systemd update that deleted the container routes on tens of thousands of nodes. The remediation list ends with an audit of the infrastructure "for potential similar legacy update channels". A vendor shipping into customer hosts inherits this problem twice: once for its own fleet, and once for every customer machine where its software is one of several agents competing to manage the same kernel state.

Figure 3 · The lifecycle that makes a remote upgrade reversible

The asymmetry is the design: only the experiment unit carries a timeout, an OnFailure edge back to stable and an ExecStopPost that forces a failed state. Doing nothing returns the host to the version it had. Source: datadog-agent.service.tmpl.

Diagram source

The decisions that matter

Six forks, each visible in the repository with a date, and the condition under which the other branch is the right one.

Decision: when you rewrite the core, do you also break the plugin contract?

Chosen

Rewrite the Agent core in Go for version 6 (February 2018) and embed CPython behind a C89 shim so existing Python checks keep running unchanged
Support both Python 2 and Python 3 backends selected at runtime by dlopen

Rejected

A clean Go-only check interface, which would have invalidated every community and customer check at once
Rejection is inferred from the shim's existence and its stated purpose, not from a written decision record

Flips when

The plugin authors are all inside your company, or the plugin count is small enough to port in one release
Datadog's was 102 integrations at the time and 401 today, which is the number that made the shim cheaper than the migration

Decision: a remote channel that changes behaviour, or one that only reads?

Chosen

A change channel, shipped December 2021, verified with Uptane rather than transport security alone
Configuration changes themselves get the experiment and promote lifecycle, not just binaries

Rejected

Plain signed-payload-over-TLS, which authenticates the connection rather than the instruction

Flips when

The channel can only pull data the host already exposes. The moment it can alter execution, compromise of your own delivery infrastructure becomes a customer-side incident, and TUF-style role separation is the cheapest defence available

Decision: how does a remote upgrade end if nobody is watching?

Chosen

It expires. The experiment unit runs under timeout with a 3000 second limit, declares OnFailure=datadog-agent.service, and forces a failed state on stop so the revert always fires
Promotion is an explicit, separately signed task

Rejected

Install and watch, with rollback as an operator action. On a fleet you cannot reach, "roll back" is a support ticket, not an operation

Flips when

You can reach the host. Inside your own fleet, a supervised rollout with a rollback runbook is cheaper than a state machine. The expiry design pays for itself precisely when nobody on your side has a shell

Decision: where do you find out whether the change works on this host?

Chosen

On the host, before committing. Preflight mode (August 2026) starts the new data plane for 90 seconds, pushes one throwaway metric through the whole path, stops it, and reports what went wrong as telemetry

Rejected

Staging and canaries alone. The design note is blunt about why: the previous way to find out was to enable the feature on a customer, "which is the worst possible time to discover an OS, permissions, proxy or packaging problem"

Flips when

Your target population is homogeneous enough that a canary is representative. It never is when the population is other people's operating systems, container runtimes, proxies and security tooling

Two further decisions are worth stating without the full block, because both are widely got wrong. The first: resource limits belong in CI, not in the incident review. Datadog's regression detector landed in December 2022 and the named quality gates in October 2024; today a change that pushes an idle Agent past 179 MiB of total memory does not merge. This is the only mechanism in the set that prevents a bad change rather than reverting one, and it is the only one that works on the versions of the software already installed, because it stops the bad version from being built. The second: the escape hatch is a feature and needs a test. Issue #43052, opened November 2025 and still open, describes a host with remote_configuration: enabled: false where the installer service crash-loops with "remote config is required to create the updater" and exit code 255. Every mechanism in this guide assumes the customer can decline it. On at least one configuration, declining produces a broken service.

Figure 4 · Choosing a change mechanism for unreachable software

Terminal nodes are mechanisms, not advice. The left branch is where most vendors stop, and it is the branch that produces a long tail of versions you can never change.

Diagram source

Decision	Chosen, and when	Rejected	Because	Evidence
Core language	Go rewrite with embedded CPython, 2018-02	Go-only check API	Preserving the check ecosystem was worth a C89 shim	rtloader README
Old major line	Ship 6 and 7 in parallel; 62 minor releases on the 6 line to 2025-01	Hard cutover at Agent 7	The installed base upgrades on its own schedule	tags, counted 2026-09-12
Change channel	Uptane-verified remote configuration, 2021-12	TLS-only config poll	The channel can alter execution on customer machines	uptane/client.go
Remote upgrade	Experiment slot that expires after 3000s, 2024-04	Install and watch	Nobody on the vendor side can log in to revert	unit template
Footprint	Hard CI bounds on memory and CPU, gates named 2024-10	Measuring footprint in production	The budget belongs to the customer's machine	quality_gate_idle
Validation	Per-host preflight rehearsal, 2026-08	Canary populations only	Customer hosts are not a sample of anything	preflight design note
Host command execution	Allowlist owned by the host administrator, fail closed	Vendor-side allowlist	The host owner has to be able to bound what runs	allowlist example
Open standard	Ship a Datadog distribution of the OTel Collector and rewrite the user's config to add its components, with a documented off switch	Proprietary protocol only	Customers arrive with OTel pipelines already running	converter README

What broke in production

Four accounts, three failure classes. Two are Datadog's published incidents, one is a defect class argued out in three other projects' issue trackers, and one is the slow failure nobody writes a postmortem for.

The classes are worth naming before the cards, because they are what transfers. Class A, unscheduled change: something alters the host without going through your release process. Class B, contested kernel state: two components manage the same resource and neither owns it. Class C, the unreachable tail: a population of installations you can no longer change, failing quietly on a clock set by someone else. Datadog's public record contains all three, and the first two share a mechanism.

Figure 5 · The March 2023 failure path, as published

Notice where the vendor's change process is: nowhere. Every arrow before the outage was drawn by a package repository and an init system. Source: Datadog root cause, 16 May 2023.

Diagram source

Postmortem

Class A: the update channel nobody owned

AssumptionChanges to the fleet arrive through the fleet's own release process.

What happenedOn 8 March 2023 at 06:00 UTC a security update to systemd was applied automatically through a legacy channel; systemd-networkd then "forcibly deleted the routes managed by the Container Network Interface (CNI) plugin (Cilium)".

Blast radiusFive regions across three cloud providers; tens of thousands of nodes between 06:00 and 07:00; more than 50 engineers within the first hour, 500 to 750 across almost 100 workstreams; full resolution 10 March at 06:25 UTC.

FixDisable the legacy update channel; configure systemd-networkd to leave the routing table unchanged on restart; audit for other channels of the same kind.

Design ruleEnumerate every mechanism that can change a host without your approval, and treat that list as a reliability control with an owner. Cloud and region diversity buy you nothing against it, because the same package lands everywhere at once.

SourceDatadog root cause, 2023-05-16, response deep dive, 2023-06-01

Source

Class B: two owners for one routing table

AssumptionSetting ManageForeignRoutes=no means systemd will not touch routes it did not create.

What happenedsystemd issue #25451, filed November 2022, reports routes added with ip route add disappearing after a networkd restart. It was closed and labelled "not-a-bug". Bottlerocket #3436 (September 2023) reports CNI rules lost the same way; k3s #7736 (June 2023) reports Cilium networking broken by an install script on systemd 249 and later. A genuine defect in the same area, routes removed despite KeepConfiguration=yes, was fixed upstream in March 2025.

Blast radiusFour organisations independently, including the one whose outage cost it two days.

FixExplicit configuration naming the owner of the state, and, upstream, two and a half years of argument before a fix landed.

Design ruleWhen two components manage the same kernel state, the interaction is undefined until someone writes down who owns it. A closed "not-a-bug" issue in a dependency's tracker is a warning about your architecture, not a resolved ticket.

Sourcesystemd #25451, Bottlerocket #3436, k3s #7736, systemd #36756

Postmortem

Class B again, this time it was Datadog's code

AssumptionAttaching eBPF programs for security monitoring is additive and does not disturb the host's networking.

What happenedA race condition caused Workload Protection's traffic-control filters to disrupt Cilium's, "breaking pod connectivity" until a manual restart. Datadog's January 2026 account also names the general hazard: hooks such as raw_syscalls tracepoints can affect production systems "from slowing connection acceptance to consuming host resources".

Blast radiusNot quantified publicly. Pod connectivity on affected hosts, recovered by restart.

FixRefuse to start with an actionable error when a mandatory program fails to load; run self-tests at startup and report them to the backend; strict memory and CPU limits; drop up to 95 percent of captured events in the kernel; support kernels down to 4.15.

Design ruleSoftware that loads code into someone else's kernel needs a startup self-test whose result leaves the machine, because the only alternative is learning from the customer that you broke their cluster.

SourceDatadog engineering, 2026-01-07

Source

Class C: the tail dies of certificate expiry, not of bugs

AssumptionAn old version stops mattering once the new one ships. Datadog's documentation asks customers to update "with every minor and patch release, or, at a minimum, monthly".

What happenedAgent 5 last had a functional release in October 2020. Version 5.32.9, in January 2023, exists only because the RPM needed re-signing with a new key. In 2025 and 2026 the repository gained certificate-authority rotation runbooks, then variants for RHEL 5 and 6 and Windows Server 2008 R2, because the embedded certificate bundle in those installs will not verify Datadog's new intake certificates.

Blast radiusUnknown and unknowable from outside: the failure mode is a host that silently stops reporting. Datadog has published no count of Agent 5 installations.

FixScripts written to the lowest interpreter versions in the install base, plus a fallback to the operating system's certificate store, plus an option for hosts that cannot even reach GitHub.

Design ruleBudget for cryptographic rotation on the oldest platform you still accept data from, and write the runbook while someone still remembers the code. Trust anchors, signing keys and TLS versions are the clock that kills frozen software.

SourceAgent v5 certificate runbook, dd-agent CHANGELOG

One more failure belongs here that has no card, because it never reached production, and that is the point. Pull request #29560 proposed FIPS support for the Agent in September 2024. It was closed unmerged that November after a reviewer observed that running the FIPS module self-tests from the entrypoint script would fail on read-only filesystems and for users without root, and would not run at all if the default entrypoint were overridden, and after the regression detector measured a 6.22 percent memory increase in one experiment. Both objections are the same objection: the change assumed facts about the host that the vendor does not get to assume. A review culture that can articulate that, and a CI system that can measure it, is worth more than any rollback mechanism.

Numbers you can plan against

Everything quantitative in this guide, with the date it was true and where it came from. Counts marked "counted" were derived from the repository on 12 September 2026 and can be reproduced with the commands in section 08.

Metric	Value	Context	As of	Source
Idle Agent memory ceiling	179 MiB	CI gate on total PSS, default config, no workload; allotment set 30 percent higher	2026-09	quality_gate_idle
Memory ceiling, all features	537 MiB	Same gate family, every product enabled	2026-09	quality_gate_idle_all_features
Experiment lifetime	3000 s	Remotely started Agent version, killed 15 s after the timeout if not promoted; 50 minutes, derived	2026-09	unit template
Preflight rehearsal window	90 s	Minimum, clamped upward; runs once per Agent start	2026-08	preflight design note
Core Agent binary	236 to 103 MiB	56 percent reduction; security agent fell 77 percent	2026-02	Datadog engineering
Installed package	1.22 GiB to 688 MiB	Uncompressed; 265 MiB to 149 MiB compressed	2026-02	Datadog engineering
Oldest supported kernel	4.15	Workload Protection, with fallbacks to 4.14 on some distributions	2026-01	Datadog engineering
Events discarded in kernel	up to 95%	Before crossing into user space, to bound host cost	2026-01	Datadog engineering
Integrations maintained	401	Counted as directories with a manifest; 102 in 2018, 196 in 2022, 260 in 2024	2026-09	integrations-core
Integrations ever removed	19	Manifest deletions in the whole history, including docker_daemon, mesos and hdfs	2026-09	integrations-core
Minor releases	130	X.Y.0 tags since 6.0.0; 9 per year in 2018, 20 in 2024, monthly in 2026	2026-09	datadog-agent tags
Old-line overlap	7 years	Agent 6 shipped 2018-02-26, last 6.x minor 2025-01-10, 62 minors in between	2026-09	datadog-agent tags
Crash-class fixes shipped	76 / 61 / 17	Occurrences of panic, crash and memory leak in the released changelog	2026-09	CHANGELOG.rst
Nodes lost, March 2023	tens of thousands	Between 06:00 and 07:00 UTC, five regions, three cloud providers	2023-03	Datadog root cause
Responders, March 2023	500 to 750	Across almost 100 workstreams; more than 50 in the first hour	2023-03	Datadog engineering
Backend ingest	100 trillion events/day	Husky, with billions of queries per day; context for what the Agent feeds	2025-10	Datadog engineering

Read these carefully

The CI ceilings, the experiment timeout and the counts are measured facts about code in the repository on 12 September 2026, and they change without announcement; re-derive them rather than quoting this page in a year. The binary sizes, kernel floor and 95 percent drop rate are Datadog's own figures with no independent measurement available. The incident numbers come from Datadog's account of its own outage, which is the only public source. Nobody has published the size of the Agent 5 install base, the fraction of hosts with remote updates enabled, or the failure rate of remote upgrade experiments, and those three unknowns are exactly where an architect copying this design carries risk.

The evidence wall

Every source behind this page, graded. Filter by kind. The mix is deliberately heavy on code and light on prose, for the reason given at the end.

Postmortem Datadog2023-05

2023-03-08 incident: root cause

The only full public root cause Datadog has published. An automatically applied systemd security update caused systemd-networkd to delete the CNI-managed routes on tens of thousands of nodes across five regions and three cloud providers.

Carry forwardMulti-cloud is not a partition against a package update that lands everywhere at once.

datadoghq.com/blog/2023-03-08-multiregion-infrastructure-connectivity-issue

Postmortem Datadog2023-06

A deep dive into our incident response

The human side of the same outage: more than 50 engineers in the first hour, almost 100 workstreams, 500 to 750 people in shifts, and a company-wide response for nearly 48 hours after a failure that took under an hour to arrive.

Carry forwardRecovery cost scales with the number of independent stacks you must repair, not with the duration of the trigger.

datadoghq.com/blog/engineering/2023-03-08-deep-dive-into-incident-response

Postmortem Datadog2026-01

Hardening eBPF for runtime security

An incident review inside an engineering post: Datadog's traffic-control filters raced with Cilium's and broke pod connectivity. Also states the guardrails, the 4.15 kernel floor and the in-kernel drop rate.

Carry forwardStartup self-tests whose results leave the machine are the only way to learn that your kernel code broke a host you cannot see.

datadoghq.com/blog/engineering/ebpf-workload-protection-lessons

Source Datadog2026-09

datadog-agent.service.tmpl

The whole reversibility design in twenty lines. Only the experiment branch of the template sets EXPERIMENT_TIMEOUT=3000s, wraps the binary in timeout, declares OnFailure=datadog-agent.service and forces a failed state with ExecStopPost=/bin/false.

Carry forwardMake the new version the one that expires, and inaction becomes a safe rollback.

raw.githubusercontent.com/DataDog/datadog-agent/main/pkg/fleet/installer/packages/embedded/tmpl/datadog-agent.service.tmpl

Design doc Datadog2026-08

Agent Data Plane preflight mode

A design note written for maintainers, not readers: why the pre-flight exists, the five conditions that gate it, why the 90 second floor is a contract, and why it is skipped when a secret backend is configured.

Carry forwardIf you cannot test on representative hosts, test on the actual host and report the result before you change anything.

raw.githubusercontent.com/DataDog/datadog-agent/main/docs/dev/agent-data-plane.md

Design doc Datadog2026-09

Installer package hooks

Documents the ordinary deb and rpm lifecycles, then the OCI-only "experiments" path: start writes the new version while keeping the old files, stop deletes the new, promote deletes the old. The hook names are the state machine.

Carry forwardTwo slots on disk with an explicit promote step turn an upgrade into a decision you can defer.

raw.githubusercontent.com/DataDog/datadog-agent/main/pkg/fleet/installer/packages/README.md

Source Datadog2021-12

Uptane client for remote configuration

Remote configuration verifies TUF metadata using Uptane, the framework built for shipping software to vehicles, with separate director and config roots and per-org stores.

Carry forwardA channel that can change execution on other people's machines needs signed instructions, not just an authenticated connection.

raw.githubusercontent.com/DataDog/datadog-agent/main/pkg/config/remote/uptane/client.go

Source Datadog2026-09

Installer default package list

The remote catalogue: the Agent, the OTel distribution and tracing libraries for eight languages, each gated on release state, site and whether the host opted into remote updates. Also carries pinned versions for CentOS 6 with the note that nothing is tested there.

Carry forwardStaged exposure is a data structure, not a process: encode who may receive a version in the code that resolves it.

raw.githubusercontent.com/DataDog/datadog-agent/main/pkg/fleet/installer/default_packages.go

Source Datadog2026-09

quality_gate_idle experiment

A CI experiment that runs the Agent with default configuration and no workload and fails the build above 179 MiB of total PSS, five intake connections, or 0.8 MiB of received traffic.

Carry forwardExpress the footprint you promise a customer as a build failure, not as a performance goal.

raw.githubusercontent.com/DataDog/datadog-agent/main/test/regression/cases/quality_gate_idle/experiment.yaml

Source Datadog2026-09

rtloader README

The compatibility shim that made the Go rewrite survivable: a C89 interface over CPython that picks its Python backend at runtime with dlopen. The line "Python2 isn't supported anymore" is the end of a migration that began in 2019.

Carry forwardRewrites are affordable when the plugin ABI survives them; price the shim against the ecosystem you would otherwise strand.

raw.githubusercontent.com/DataDog/datadog-agent/main/rtloader/README.md

Source Datadog2024-10

Commit: rtloader, drop python2 support

One commit, 15 October 2024, removes the Python 2 backend. Python 2 reached upstream end of life in January 2020 and Agent 7 shipped in December 2019, which puts the length of the compatibility tail at nearly five years.

Carry forwardDate your deletions: the gap between "we shipped the replacement" and "we deleted the old path" is the real migration cost.

github.com/DataDog/datadog-agent/commit/4b8b2b5ab9

Source Datadog2026-03

dd-agent: legacy OS scripts for CA rotation

A March 2026 commit on the Agent 5 repository adding bash 3.1 compatible scripts for RHEL 5 and 6 and PowerShell 2.0 compatible scripts for Windows Server 2008 R2, including an insecure download fallback for hosts whose CA bundle cannot verify GitHub.

Carry forwardThe oldest host in your install base sets the floor for every tool you will ever need to run on it.

github.com/DataDog/dd-agent/commit/2330e07b

Source Datadog2026-07

dd-agent README, still supported

The Agent 5 repository states the line is "still supported" and documents a development setup requiring Python 2.7. Its last touch is July 2026, eight years after the successor shipped.

Carry forward"Supported" is a promise about your future engineering time; count the versions you have made it for.

raw.githubusercontent.com/DataDog/dd-agent/master/README.md

Source Datadog2023-01

dd-agent CHANGELOG, release 5.32.9

A release of a frozen product whose entire content is a re-signed RPM and two dependency bumps. The reason a dead line still cuts releases is almost always a key, a certificate or a repository signature.

Carry forwardKeep a build pipeline alive for every version you still accept data from, because you will need to re-sign it.

raw.githubusercontent.com/DataDog/dd-agent/master/CHANGELOG.md

Source Datadog2024-11

PR #29560: FIPS support, closed unmerged

Closed after review: FIPS self-tests in the entrypoint would fail on read-only filesystems and without root, and would be skipped when the entrypoint is overridden. The regression detector separately flagged a 6.22 percent memory increase.

Carry forwardReview a change to fleet-resident software against the least privileged, most locked-down host in the population.

github.com/DataDog/datadog-agent/pull/29560

Source Datadog2025-11

Issue #43052: the installer needs the channel you turned off

On Agent 7.72.2 with remote configuration disabled, the installer service fails with "remote config is required to create the updater" and exit code 255, restarting in a loop. Open as of September 2026.

Carry forwardTest the configuration where the customer declines your control channel; it is the one that never appears in a demo.

github.com/DataDog/datadog-agent/issues/43052

Source Datadog2026-06

PR #50607: error logs to internal telemetry

Adds experimental forwarding of Agent ERROR logs so Datadog engineers can see failure patterns across the fleet. Off by default, with the message text dropped before send and the commit SHA included so a stack location can be resolved without the text.

Carry forwardYou can observe a fleet you do not own by shipping identifiers rather than content.

github.com/DataDog/datadog-agent/pull/50607

Source Datadog2026-09

PowerShell check allowlist

The example file states the rule: the allowlist must live in a directory writable only by Administrators and SYSTEM, must be owned by them, and the check "refuses to load it and collects nothing (fail closed)" otherwise.

Carry forwardWhen your software executes host commands, make the host's administrator the author of the policy and fail closed without one.

raw.githubusercontent.com/DataDog/datadog-agent/main/cmd/agent/dist/protected/powershell_allowlist.yaml.example

Design doc Datadog2026-09

OpenTelemetry converter component

"The converter enhances the user provided configuration": Datadog's collector distribution rewrites the customer's OTel config to insert its own processors and extensions, with each feature individually disablable and a documented way to turn them all off.

Carry forwardIf you modify a configuration the customer owns, enumerate every modification as a named feature with an off switch.

raw.githubusercontent.com/DataDog/datadog-agent/main/comp/otelcol/converter/README.md

Source systemd2022-11

systemd #25451, closed as not-a-bug

Routes added outside networkd disappear on restart despite ManageForeignRoutes=no. Filed November 2022, labelled "not-a-bug", four months before the same mechanism removed Cilium's routes across Datadog's fleet.

Carry forwardSearch your dependencies' trackers for the mechanism, not the symptom; a closed issue there is an unowned risk in your design.

github.com/systemd/systemd/issues/25451

Source systemd2025-03

systemd #36756, the fix

Merged March 2025: routes on an interface could be removed on restart even with KeepConfiguration=yes, traced to an earlier change, with tests added.

Carry forwardUpstream fixes to shared-state bugs arrive years after the incidents; your architecture has to survive the interval.

github.com/systemd/systemd/pull/36756

Source Bottlerocket2023-09

Bottlerocket #3436

On a Kubernetes-focused OS, restarting systemd-networkd removes the IP rules installed by the VPC CNI, leaving only the defaults. Same class, different organisation, six months after Datadog's outage.

Carry forwardTwo independent reports of the same interaction make it a design constraint rather than an anomaly.

github.com/bottlerocket-os/bottlerocket/issues/3436

Source k3s2023-06

k3s #7736

An install script that saves and restores iptables rules breaks Cilium on systemd 249 and later. The workaround is to skip the script's start step entirely.

Carry forwardInstallers that touch global host state need an explicit "do not manage this" flag, or every CNI becomes your problem.

github.com/k3s-io/k3s/issues/7736

Eng blog Datadog2026-02

Reducing Agent Go binary sizes by up to 77%

Concrete numbers for every binary and the package as a whole, and an unusually direct statement of why size matters when the artefact lands on other people's machines: network costs, resource usage, and constrained platforms.

Carry forwardDistribution size is a customer-facing cost, and it grows silently until someone measures it per platform.

datadoghq.com/blog/engineering/agent-go-binaries

Eng blog Datadog2022-05

Introducing Husky

The backend counterpart: a third-generation event store on commodity object storage, built because one tenant's burst degraded queries for everyone sharing a shard. The migration took over eighteen months.

Carry forwardThe vendor's own multi-tenancy problem and the customer's noisy-neighbour problem are the same problem at two scales.

datadoghq.com/blog/engineering/introducing-husky

Eng blog Datadog2025-10

Inside Husky's query engine

Scale and cache behaviour for the system the Agent feeds: more than 100 trillion events and billions of queries a day, with 80 percent result-cache and 70 percent range-cache hit ratios.

Carry forwardSizing the ingest side tells you what a one percent change in agent-side sampling is worth.

datadoghq.com/blog/engineering/husky-query-architecture

Vendor Datadog2026-09

Agent version differences

States the upgrade expectation ("every minor and patch release, or, at a minimum, monthly") and confirms Agent 6 as "a complete rewrite of the core Agent in Golang". The gap between that expectation and the Agent 5 runbooks is the subject of this guide.

Carry forwardWrite down the upgrade cadence you expect, then measure the distribution you actually have.

docs.datadoghq.com/agent/guide/version_differences

Vendor Datadog2026-09

Fleet Automation documentation

The customer-facing framing of the installer daemon: govern and remotely manage Agents and OTel Collectors, identify outdated versions and upgrade them, with the capability unavailable on some government sites.

Carry forwardThe remote update capability is a product surface with its own regional and regulatory exclusions; plan for the hosts it will never reach.

docs.datadoghq.com/agent/fleet_automation

Two absences shape this wall. There are no papers and no conference talks in it, because this session could not reach arxiv, USENIX or video hosts; nothing in the guide rests on one, and any claim that would have needed one has been left out. And there is exactly one incident for which Datadog has published a root cause, in a decade of operating a global observability platform. For a company that sells incident management, that is a striking ratio, and it means every statement here about how Datadog's own fleet fails is either from that one incident or reconstructed from code.

Build a miniature, then productionise it

Seven rungs. The line between toy and real is at rung four, where the mechanism stops protecting you from your own mistakes and starts protecting hosts you cannot log into.

Two slots and a supervisor

Package a trivial daemon into /opt/yourthing/{stable,experiment} with two systemd units that declare Conflicts on each other. Write the start, stop and promote operations as three shell functions.

Done when: promoting swaps which directory the stable unit runs from, with no service gap longer than a restart. Teaches: the host's init system, not your code, is what guarantees only one version runs.

Make the new version expire

Wrap the experiment unit's ExecStart in timeout, add OnFailure pointing at the stable unit, and force a failed state on stop. Set the timeout to two minutes while you are learning.

Done when: you start an experiment, walk away, and the machine is running the old version when you come back. Teaches: reversibility you do not have to trigger is the only kind that works on unreachable hosts.

A signed instruction, not an authenticated connection

Put the three operations behind a poller that only acts on TUF-signed metadata, using go-tuf or python-tuf. Keep the signing keys off the serving host.

Done when: a payload served over a valid TLS connection but signed with the wrong key is refused, and the refusal is visible in telemetry. Teaches: your delivery infrastructure is inside the threat model.

Report the outcome, not the logs

Have the daemon report installed versions, the result of each operation and a single startup self-test to a collector you run. Ship identifiers and error codes, never message bodies.

Done when: you can answer "how many hosts are on which version, and how many experiments failed last week" without asking anyone. Teaches: the fleet inventory is the product; without it every other mechanism is blind.

Rehearse on the target host

Before enabling a new component for real, start it for ninety seconds, push one synthetic item through the full path, stop it, and report what failed. Skip the rehearsal when secrets are configured, as Datadog does, so nothing resolves a secret to disk.

Done when: a host with a broken proxy configuration is reported as unsuitable before you ever enable the component there. Teaches: heterogeneous populations cannot be sampled; they have to be interrogated one at a time.

Put the customer's budget in CI

Add a load harness that runs your daemon idle and under a representative workload, and fail the build above a fixed RSS and CPU bound. Write the bound as a number in a file, with the allotment set above it.

Done when: a pull request that adds 30 MiB of resident memory fails before review, and the failure names the experiment. Teaches: you cannot roll back a footprint regression that has already been installed, so it must never build.

Inventory every other channel that can change the host

For one representative host class, list everything that can alter it without your release process: unattended upgrades, vendor agents, configuration management, image rebuilds, CA and key rotations. Give each an owner and an alert when its state changes.

Done when: the list exists, has owners, and one item on it has been disabled or pinned as a result. Teaches: the change you did not schedule is the one that arrives everywhere simultaneously.

Keep hunting

This guide was built almost entirely with git and GitHub search, because the repository is the least edited account a company publishes. These are the commands and queries that produced it.

Dating an architecture from a clone

git clone --filter=blob:none <repo>
git log --reverse --date=short --format='%ad %s' -- <path> | head -2
git log --diff-filter=D --date=short --format='%ad %s' -- <path>
git log --date=format:%Y --format='%ad' | sort | uniq -c
git for-each-ref --format='%(creatordate:short) %(refname:short)' refs/tags

Finding the argument, not the announcement

is:pr is:closed is:unmerged sort:comments-desc repo:<org>/<repo>
repo:<org>/<repo> "is required" OR "refuses to start" in:body is:issue
path:test/regression OR path:**/quality_gate* upper_bound
path:**/*.service.tmpl OnFailure timeout

Reading an install base

grep -rl "deprecat" releasenotes/notes | wc -l
git log --format='%ad' --date=format:%Y -- . | sort | uniq -c # on the archived repo
git ls-tree -r $(git rev-list -1 --before=YYYY-MM-DD HEAD) --name-only | grep -c manifest.json
"root ca" OR "certificate" rotation runbook repo:<org>/<legacy-repo>

Corroborating a postmortem in someone else's tracker

<mechanism named in the postmortem> repo:<dependency> is:issue
"not-a-bug" OR "works as intended" <mechanism>
<mechanism> site:github.com "restart" routes OR rules deleted

References

Datadog, 2023-03-08 incident: root cause Datadog blog, 16 May 2023. Checked 2026-09-12.
Datadog, 2023-03-08 incident: a deep dive into our incident response Datadog engineering blog, 1 June 2023. Checked 2026-09-12.
Datadog, Hardening eBPF for runtime security Datadog engineering blog, 7 January 2026. Checked 2026-09-12.
Datadog, How we reduced the size of our Agent Go binaries by up to 77% Datadog engineering blog, 18 February 2026. Checked 2026-09-12.
Datadog, Introducing Husky Datadog engineering blog, 17 May 2022. Checked 2026-09-12.
Datadog, Inside Husky's query engine Datadog engineering blog, 1 October 2025. Checked 2026-09-12.
Datadog, Agent version differences Datadog documentation. Checked 2026-09-12.
Datadog, Fleet Automation Datadog documentation. Checked 2026-09-12.
datadog-agent, systemd unit template GitHub, main branch. Checked 2026-09-12.
datadog-agent, installer package hooks GitHub, main branch. Checked 2026-09-12.
datadog-agent, installer default package list GitHub, main branch. Checked 2026-09-12.
datadog-agent, Uptane client GitHub, main branch; first commit 17 December 2021. Checked 2026-09-12.
datadog-agent, Agent Data Plane preflight mode GitHub, main branch; added 11 August 2026. Checked 2026-09-12.
datadog-agent, quality_gate_idle experiment GitHub, main branch. Checked 2026-09-12.
datadog-agent, quality_gate_idle_all_features experiment GitHub, main branch. Checked 2026-09-12.
datadog-agent, rtloader README GitHub, main branch. Checked 2026-09-12.
datadog-agent, OpenTelemetry converter component GitHub, main branch. Checked 2026-09-12.
datadog-agent, PowerShell allowlist example GitHub, main branch. Checked 2026-09-12.
datadog-agent, CHANGELOG GitHub, main branch; 7.83.1 released 9 September 2026. Checked 2026-09-12.
datadog-agent, commit: rtloader, drop python2 support GitHub, 15 October 2024. Checked 2026-09-12.
datadog-agent, PR #29560: Add FIPS support to the Agent GitHub, opened 25 September 2024, closed unmerged 7 November 2024. Checked 2026-09-12.
datadog-agent, PR #50607: forward Agent error logs to internal telemetry GitHub, June 2026. Checked 2026-09-12.
datadog-agent, issue #43052: Failed to start datadog-agent-installer.service GitHub, opened 13 November 2025. Checked 2026-09-12.
datadog-agent repository GitHub. Tag and commit counts derived 2026-09-12.
integrations-core repository GitHub. Integration counts derived from the tree 2026-09-12.
dd-agent README (Agent 5) GitHub, master branch. Checked 2026-09-12.
dd-agent CHANGELOG GitHub, master branch; 5.32.9 released 12 January 2023. Checked 2026-09-12.
dd-agent, Agent v5 certificate update runbook GitHub, master branch. Checked 2026-09-12.
dd-agent, commit: legacy OS scripts for sectigo CA rotation GitHub, 10 March 2026. Checked 2026-09-12.
systemd, issue #25451 GitHub, opened 20 November 2022, closed as not-a-bug. Checked 2026-09-12.
systemd, PR #36756 GitHub, merged 15 March 2025. Checked 2026-09-12.
Bottlerocket, issue #3436 GitHub, opened 8 September 2023. Checked 2026-09-12.
k3s, issue #7736 GitHub, opened 11 June 2023. Checked 2026-09-12.