Replacing the package with the image: ten years of Red Hat changing the unit of change
How Red Hat moved the unit of change for a fleet of machines from the RPM transaction to the versioned filesystem tree to the OCI container image between 2016 and 2026, which of its own host products it killed to get there, and what each step pushed outside the atomic artefact.
Four generations of one host operating system, reconstructed from the artefacts the decisions were taken in: Fedora CoreOS design records and tracker issues, OpenShift enhancement proposals with their rejected alternatives, a proposal that died of inactivity and returned three years later, and the git history of the clients. A reader leaves able to say which of a machine's inputs their atomic artefact actually contains, which inputs will skew and when that skew becomes visible, what each customization escape hatch costs in support surface, and why a canary population expressed as a percentage of the fleet is not a test matrix.
The container registry was not an idea Red Hat lacked in 2018: it was considered in tracker issue #23, declined as the default on the stated ground that "OCI today is that there's no deltas", and the format chosen instead, rojig, was deleted from rpm-ostree in May 2021 along with its design document, while experimental container encapsulation had already existed in the same codebase since August 2017.
What you get out of it
- The delivery format was settled by the tooling customers already operate rather than by transport efficiency: the 2018 objection to OCI, that it has no deltas, was never answered, it was outvoted.
- Every generation re-opened a customization hatch it had closed, in 2020 as release-versioned extensions, in 2022 as customer-derived images, in 2025 as on-cluster builds, and the 2022 enhancement states the price plainly: the node OS version is decoupled from the cluster release.
- Provision-once configuration became a permanent constraint: OpenShift's 2025 enhancement records that install-time Ignition fields lock a cluster for its whole life and that the only previous remedy was to re-provision it.
- The most expensive artefact in the record is the one nobody owned: updating a cluster's boot images was proposed in February 2020, closed unmerged two years later in an argument about which operator owned it, re-proposed in 2023, and is still opt-in in July 2026, with seven named classes of scale-up bug attributed to the skew.
- Generation four's own documentation says the automatic rollback net is incomplete on its newer composefs backend, which has no boot-complete service and no bootloader entry counting, so the central promise of an image-based system rests on a detector that is not there yet.
Scope
Why this, now. In August 2026 the maintainer of Fedora CoreOS proposed merging it with the Fedora bootc base images, citing the 2019 merger of Atomic Host and Container Linux as the precedent, which makes the whole decade legible as one sequence rather than four products.
What it does not cover. Conventional RHEL with dnf, workload updates inside Kubernetes, the equivalent immutable-host designs at SUSE, Canonical and Google, and desktop variants beyond their appearance in the adopter list. It also contains no vendor blog post, conference talk, paper or independent benchmark, because this session's egress policy allowed code hosts only; the four incident entries are issue-tracker reports with reproductions rather than published post-incident reviews, and product milestone dates are therefore given as commits and merges rather than as release announcements.
Other field guides
Everything they deleted was on the inside: ten years of HashiCorp, read from its own repositories
HashiCorp spent a decade deleting things from the inside of its products: a Raft log store, a dependency cluster, a process on every node, four produ…
24 sources · 5 organisations · 1 postmortemKeeping the main branch green: thirteen years of merge queues, read from the repositories that ran them
The merge-queue pattern traced through its primary record: Rust's three generations of bors, Zuul's speculative gating, Kubernetes' Tide, GitLab's tr…
20 sources · 11 organisations · 2 postmortemsReplacing the engine room in public: ten years of Sentry, read from its RFCs, releases and self-host floor
A decade of component replacement inside one product, measured in the artefact other people install: seven containers at the 9.1.2 tag, thirty by the…
32 sources · 3 organisations · 5 postmortems