Evidence ledger
30 sources
Checked 07 Oct 2026
Evidence ledger
One row per claim in Replacing the package with the image: ten years of Red Hat changing the unit of change: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Field guide: Replacing the package with the image: ten years of Red Hat changing the unit of change, read from its own trackers and enhancements
Research date: 2026-10-07. Every URL below was fetched in this session.
Network constraint on this hunt. The session's egress policy allowed github.com,
raw.githubusercontent.com and gitlab.com and refused everything else, including
blog.cloudflare.com, arxiv.org, www.usenix.org, datatracker.ietf.org,
developers.cloudflare.com, docs.redhat.com, fedoraproject.org and youtube.com. The
ledger is therefore built from code hosts only: repositories, in-repository design documents,
issue threads, pull requests and git history. There are no engineering blog posts, conference
talks, papers or independent benchmarks in it, and the "postmortem" rows are issue-tracker
incident reports rather than published post-incident reviews. Four rows are derived by counting
git history locally, and are labelled as derived.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Fedora / Red Hat | fedora-coreos-tracker README | vendor | 2018-07 onward | 2026-10-07 | https://github.com/coreos/fedora-coreos-tracker | Fedora CoreOS was created as the merge of two existing host systems, one of them acquired | "It aims to combine the best of both CoreOS Container Linux and Fedora Atomic Host, integrating technology like Ignition from Container Linux with rpm-ostree and SELinux hardening from Project Atomic." |
| 2 | Fedora / Red Hat | fedora-coreos-tracker PRD.txt | adr | added 2018-08-17 | 2026-10-07 | https://github.com/coreos/fedora-coreos-tracker/blob/main/PRD.txt | The successor relationship was stated as product requirement, not retrospective framing | "The demand for a container focused operating system that is automatically updated has been proven by Container Linux and Atomic Host. Fedora CoreOS is the successor to these distributions." |
| 3 | Fedora / Red Hat | fedora-coreos-tracker Design.md, "OSTree Delivery Format" | adr | added 2018-08-23, last touched 2025-06-16 | 2026-10-07 | https://github.com/coreos/fedora-coreos-tracker/blob/main/Design.md | Three delivery models were on the table in 2018; the container registry was deferred, not unknown | "There are three proposed delivery models ... OSTree Repo ... rojig ... OCI: OSTree commits are packaged up in OCI container images and delivered via a container registry. Currently the plan in Fedora CoreOS is to deliver content via a plain OSTree Repo and augment our strategy with either rojig or OCI if it proves useful or necessary." |
| 4 | Fedora / Red Hat | fedora-coreos-tracker issue #23, delivery model | adr | 2018-08-03, closed, labelled status/decided | 2026-10-07 | https://github.com/coreos/fedora-coreos-tracker/issues/23 | The stated argument for the registry was the customer's existing tooling; the stated argument against was transport efficiency | cgwalters: "a metric ton of tools...know how to mirror container images", offline usage is "absolutely critical"; against: "OCI today is that there's no deltas". Recommendation was both rojig and oscontainer, "with rojig as the default". |
| 5 | Fedora / Red Hat | fedora-coreos-tracker Design.md, "Release Streams" | adr | added 2018-08-23 | 2026-10-07 | https://github.com/coreos/fedora-coreos-tracker/blob/main/Design.md | The canary population is a documented fraction of the user's own fleet, and the cadence is explicitly non-contractual | "Users will be encouraged to run most of their production systems on stable, and a few percent of their systems on each of next and testing to catch regressions before they reach stable." and "will initially have two weeks between releases" |
| 6 | Red Hat | rpm-ostree, first commit | source | 2013-12-21 | 2026-10-07 | https://github.com/coreos/rpm-ostree | The hybrid image/package client predates the container era by a decade | git log --reverse: first commit 2013-12-21 19:41:30 -0500 (derived locally) |
| 7 | Red Hat / GNOME | ostree, first commit | source | 2011-10-09 | 2026-10-07 | https://github.com/ostreedev/ostree | The underlying deployment model is fifteen years old | git log --reverse: first commit 2011-10-09 17:03:08 -0400 (derived locally) |
| 8 | Red Hat | rpm-ostree commit f41183e0, app/ex-container |
source | 2017-08-10 | 2026-10-07 | https://github.com/coreos/rpm-ostree/commit/f41183e0e568e74b4ea53ad4d83b7c3ff054cf20 | Experimental container encapsulation existed a year before the 2018 delivery-format decision | Commit subject "app/ex-container: Port to new style", 2017-08-10 13:39:08 +0000 |
| 9 | Red Hat | rpm-ostree commit 562e03f7, "Remove large chunks of rojig code" | source | 2021-05-18 | 2026-10-07 | https://github.com/coreos/rpm-ostree/commit/562e03f7c1d6b771f819828382e7d6b81d41f383 | The 2018 default delivery model was deleted, design document included | Deleted paths include design/rojig.md, src/app/rpmostree-compose-builtin-rojig.cxx, src/libpriv/rpmostree-rojig-*, tests/compose/test-rojig-e2e.sh |
| 10 | Red Hat | rpm-ostree README, project status note | vendor | README last modified 2025-11-25 | 2026-10-07 | https://github.com/coreos/rpm-ostree | The client that implemented generations two and three has handed new work to generation four | "Currently, development focus has shifted to bootc, dnf, and the ecosystem around those tools. ... In general, new major features related to bootable containers should land in those projects instead." |
| 11 | Red Hat | Fedora CoreOS enhancement, os/coreos-layering.md |
adr | added 2021-11-16 | 2026-10-07 | https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md | Red Hat names the decade-long tension itself, and keeps the base-image/user-content split as a goal | "Since the creation of Container Linux (CoreOS) as well as Atomic Host, and continuing into the Fedora/RHEL CoreOS days, we have faced a constant tension around what we ship in the host system." and "One goal the current CoreOS team has in this is to still preserve the distinction we have between \"base image\" and user content. This argues for a declarative input that only allows controlled mutation." |
| 12 | Red Hat | Fedora CoreOS enhancement, os/coreos-layering.md, Ignition section |
adr | added 2021-11-16 | 2026-10-07 | https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md | Moving content into the image did not remove the second configuration mechanism | "This proposal does not replace Ignition. Ignition will still play at least two key roles: Setting up partitions and storage ... Machine/node specific configuration" |
| 13 | Red Hat | OpenShift enhancement, rhcos/extensions.md |
adr | creation-date 2020-04-21, merged 2020-05-27 | 2026-10-07 | https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md | The first escape hatch was opened in 2020 and its own author recorded the risk it created | "This will blaze a trail that will make it easier to install 3rd party RPMs, which is much more of a risk in terms of compatibility and for upgrades." Alternatives rejected: "Multiple OS builds: Becomes a combinatorial nightmare." and "Do nothing: RHCOS may continue to grow with every new case". |
| 14 | Red Hat | OpenShift enhancement, rhcos/extensions.md, shipping vs managing |
adr | 2020-04-21 | 2026-10-07 | https://github.com/openshift/enhancements/blob/master/enhancements/rhcos/extensions.md | Delivery format determines the management interface, stated flatly | "How one ships software has a huge impact on how one manages the software." |
| 15 | Red Hat | OpenShift enhancement, ocp-coreos-layering.md |
adr | creation-date 2021-10-19, merged 2022-08-22, last updated 2025-04-08 | 2026-10-07 | https://github.com/openshift/enhancements/blob/master/enhancements/ocp-coreos-layering/ocp-coreos-layering.md | The motivation is the untestability of the two-mechanism node state, not customization alone | Motivation items: "It is very difficult today to roll out a hotfix (kernel or userspace)."; "There are upgrade problems today caused by configuration updates of the MCO applied separately from OS changes."; "It is difficult to inspect and test in advance what configuration will be created by the MCO without having it render the config and upgrade a node." Goal: "Gain transactional configuration+code changes by having configuration+encode captured atomically in a container." |
| 16 | Red Hat | OpenShift enhancement, ocp-coreos-layering.md, Phase 0 |
adr | 2022-08-22 | 2026-10-07 | https://github.com/openshift/enhancements/blob/master/enhancements/ocp-coreos-layering/ocp-coreos-layering.md | The OS was already shipped inside a container image before it was shipped as one | "the existing machine-os-content has an ostree repository inside a UBI image, which is hard to inspect and cannot be used for derived builds" |
| 17 | Red Hat | OpenShift enhancement, ocp-coreos-layering.md, user stories |
adr | 2022-08-22 | 2026-10-07 | https://github.com/openshift/enhancements/blob/master/enhancements/ocp-coreos-layering/ocp-coreos-layering.md | The escape hatch detaches the node's OS version from the cluster's release version | "Note that future oc adm upgrade will not override this image - the node OS version is now decoupled from the cluster/release image default." |
| 18 | Red Hat | OpenShift enhancement, machine-config/machine-config-irreconcilable-changes.md |
adr | creation-date 2025-07-07, merged 2025-09-17 | 2026-10-07 | https://github.com/openshift/enhancements/blob/master/enhancements/machine-config/machine-config-irreconcilable-changes.md | Provision-once configuration became a permanent constraint on the cluster, admitted six years after OpenShift 4 shipped | "Once the user configures install-time parameters with the initial Ignition specification used at cluster install-time, the MCO will prevent any further changes to fields that MCDs do not support (known as irreconcilable fields), locking the user into an Ignition specification for the rest of the life of the cluster." and "their only real option today would be to re-provision their cluster, which is costly and time consuming" |
| 19 | Red Hat | OpenShift enhancement, machine-config/manage-boot-images.md |
adr | creation-date 2023-10-16, last updated 2026-07-16 | 2026-10-07 | https://github.com/openshift/enhancements/blob/master/enhancements/machine-config/manage-boot-images.md | The install media is an unmanaged fleet member, and the skew produced seven named classes of production bug | "These boot image references are not updated on an upgrade, so any node scaled up using it will boot up with the original "install" bootimage. This has caused a myriad of issues during scale-up due to this version skew" followed by linked issues for Afterburn, podman, skopeo, composefs, sigstore GA, aarch64 bootloaders and secure boot / kernel certs |
| 20 | Red Hat | OpenShift enhancement, manage-boot-images.md, stub Ignition note |
adr | 2023-10-16 onward | 2026-10-07 | https://github.com/openshift/enhancements/blob/master/enhancements/machine-config/manage-boot-images.md | An earlier attempt at the same fix was merged, reverted, and then re-landed for one platform only; spec 2 configs are still being migrated | "There has been a previous effort to manage the stub Ignition config. It was reverted and then brought back just for bare metal clusters." and "On clusters installed prior to 4.6, the stub Ignition would be of the spec 2 format - which cannot be used by the newer boot images." |
| 21 | Red Hat | openshift/enhancements PR #201, "bootimages: Downloading and updating bootimages via release image" | source | opened 2020-02-04, closed unmerged 2022-02-04 | 2026-10-07 | https://github.com/openshift/enhancements/pull/201 | The boot-image problem was proposed in 2020, argued over ownership, and closed by inactivity | Reviewer: "this belong together with all the upgrading business logic to the Machine Config Operator", with concern about "scattering our upgrade process into different components". State: closed, unmerged, after repeated lifecycle closure. |
| 22 | Red Hat | openshift/machine-config-operator issue #1443 | postmortem | opened 2020-02-05 | 2026-10-07 | https://github.com/openshift/machine-config-operator/issues/1443 | A node can reach a config state the reconciler can neither apply nor undo | yuqi-zhang: "Let's say if I have a file at /home/core/test, and then I apply a new machineconfig to write to /home/core/test/test, since /home/core/test is a file, the MCO properly catches that it is unable to create a directory there, and thus degrades." After deleting the MachineConfig: "It will continuously fail-loop on Marking Degraded due to: failed to create directory ...". Recovery requires editing the node annotation by hand. |
| 23 | Red Hat | openshift/machine-config-operator issue #2635 | postmortem | closed 2022-03-28 | 2026-10-07 | https://github.com/openshift/machine-config-operator/issues/2635 | The record needed to recover from a failed update is garbage-collected during the update | Issue title: "Deleting old rendered MachineConfigs during an update makes recovery super hard" (11 comments) |
| 24 | Fedora / Red Hat | fedora-coreos-tracker issue #1465, "increase size of our /boot partition for new installs" | postmortem | opened 2023-04-12, still open | 2026-10-07 | https://github.com/coreos/fedora-coreos-tracker/issues/1465 | The install-time partition layout is not fixable on machines already running | dustymabe: "/boot" is currently 384M and "we have been hitting out of space issues"; the decision is to "limit this change to newly deployed systems (i.e. don't try to re-arrange the partition table of updating systems)". Labelled priority/high. |
| 25 | Fedora / Red Hat | fedora-coreos-tracker issue #1306 | postmortem | opened 2022-09-26, closed 2023-03-11 | 2026-10-07 | https://github.com/coreos/fedora-coreos-tracker/issues/1306 | A promoted stable release failed to boot on a platform and disk topology the canary streams did not cover | Reported against stable 36.20220906.3.2 on EC2 instances with local NVMe: "dev-nvme0n1.device: Job dev-nvme0n1.device/start timed out" and "ignition-disks.service: Main process exited, code=exited, status=1/FAILURE"; 36.20220806.3.0 worked on the same configuration. |
| 26 | Fedora / Red Hat | fedora-coreos-tracker issue #2214, "CoreOS and Image Mode/Bootc Unification" | adr | opened 2026-08-26 | 2026-10-07 | https://github.com/coreos/fedora-coreos-tracker/issues/2214 | Two overlapping host projects are being merged again, and the 2019 merge is cited as the precedent | dustymabe describes them as "similar, but separate things within the Fedora ecosystem" that are "more of a hindrance than it's worth", lists "We build differently (unintentional legacy, konflux didn't exist when CoreOS started)", proposes a working group for Fedora 46, and compares the effort to "the Atomic Host and Container Linux upstream efforts coming together". |
| 27 | Red Hat / CNCF | bootc README | vendor | repository created 2021-04-03 | 2026-10-07 | https://github.com/containers/bootc | Generation four's stated goal is the OCI image as the transport for the host, with a compatibility promise | "aims to apply the same technique for bootable host systems - using standard OCI/Docker containers as a transport and delivery format for base operating system updates"; "At runtime on a target system, the base userspace is not itself running in a \"container\" by default"; "We will ensure that every existing system can be upgraded in place seamlessly across any future changes." |
| 28 | Red Hat / CNCF | bootc docs, bootc-filesystem.7.md |
source | current at 2026-10-07 | 2026-10-07 | https://github.com/containers/bootc/blob/main/docs/src/bootc-filesystem.7.md | The newest generation demotes its predecessor's core technology to an implementation detail, and still carries a mutable /etc |
"bootc is intending to be a \"fresh, new container-native interface\", and ostree is an implementation detail."; composefs "will ensure that the entire / is a read-only filesystem which is very important for achieving correct semantics."; "The /etc directory contains mutable persistent state by default; however, it is supported (and encouraged) to enable the etc.transient config option" |
| 29 | Red Hat / CNCF | bootc docs, bootc-boot-failure-detection.7.md |
source | current at 2026-10-07 | 2026-10-07 | https://github.com/containers/bootc/blob/main/docs/src/bootc-boot-failure-detection.7.md | The rollback safety net differs by backend, and the newer backend does not have it yet | "There is also a bootc-root-setup.service that runs during initramfs ... but if this service fails, the system will not boot at all (emergency mode or hang)." and "At the current time, the composefs backend does not configure boot entry counting, this is likely to be added in the future." |
| 30 | Red Hat / CNCF | bootc docs, bootc-package-managers.7.md |
source | current at 2026-10-07 | 2026-10-07 | https://github.com/containers/bootc/blob/main/docs/src/bootc-package-managers.7.md | Shipping a new invariant means teaching every existing tool about it | "A first recommendation here is that package managers should detect if /usr is read-only, and provide a useful error message referring users to documentation guidance." with the quoted dnf and apt failures on read-only roots |
| 31 | Red Hat / CNCF | bootc ADOPTERS.md | casestudy | current at 2026-10-07 | 2026-10-07 | https://github.com/containers/bootc/blob/main/ADOPTERS.md | The format is being adopted by other distributions, including competitors, and the lineage is dated | Direct adopters listed include Red Hat (2024), Universal Blue (2024), HeliumOS (2024), AlmaLinux Atomic SIG (2025), ALT Atomic (2025), Caligra (2025), CIQ for Rocky Linux (2026), Kheeper (2026); indirect via ostree lists Endless (2014) and Red Hat (2015); "The underlying ostree project is over 13 years old (as of 2024)." |
| 32 | Red Hat | ostree-rs-ext, archive notice | source | archived 2025-01-15 | 2026-10-07 | https://github.com/ostreedev/ostree-rs-ext | The bridge library between generation three and OCI was absorbed into generation four | "NOTE: THIS REPOSITORY IS MOVED INTO https://github.com/containers/bootc/ ... The future of ostree and containers/OCI will be driven by bootc." |
| 33 | Fedora / Red Hat | Zincati README | vendor | current at 2026-10-07 | 2026-10-07 | https://github.com/coreos/zincati | The host update agent is a scheduling client, not a downloader: phased rollouts, maintenance windows and an external lock manager | Features include "Agent for continuous auto-updates, with support for phased rollouts", "Multiple update strategies for finalization/reboot", "Local maintenance windows on a weekly schedule", "Support for cluster-wide reboot orchestration, via an external lock-manager" |
| 34 | Fedora / Red Hat | Zincati docs, Cincinnati protocol | adr | current at 2026-10-07 | 2026-10-07 | https://github.com/coreos/zincati/blob/main/docs/development/cincinnati/protocol.md | Updates are distributed as a graph with per-client caution, descended from Chrome's updater protocol | "Cincinnati is a protocol to provide \"update hints\" to clients, and it builds upon experiences with the Omaha update protocol."; "Cincinnati uses a directed acyclic graph (DAG) to represent the complete set of valid update-paths."; request parameters include rollout_wariness, group and os_checksum. |
| 35 | CoreOS Inc / Red Hat | Ignition, tag history | source | first commit 2013-06-13; v0.1.0 2015-07-14; v2.0.1 2019-07-24 | 2026-10-07 | https://github.com/coreos/ignition | Two config-specification lines were maintained in parallel while the fleet moved from spec 2 to spec 3 | git log/tag dates (derived locally): spec 3 development starts 2019-01-03 ("types/v2_4_exp: rename to 3_0_exp"); v2.0.1 tagged 2019-07-24; v0.35.0 still tagged 2020-01-22 alongside v2.4.0 on 2020-07-13 |
| 36 | CoreOS Inc | container-linux-config-transpiler README | source | archived repository | 2026-10-07 | https://github.com/coreos/container-linux-config-transpiler | The configuration front end was rebuilt for each generation rather than carried across | "NOTE: This tool is for Container Linux, not Fedora CoreOS. See FCCT for the Fedora CoreOS equivalent." |
| 37 | Kinvolk / Microsoft | flatcar/Flatcar README | source | current at 2026-10-07 | 2026-10-07 | https://github.com/flatcar/Flatcar | The abandoned generation two product continues as a third-party fork with the same design claims | "Flatcar ships only the essentials needed to run containers, no package manager, no configuration drift. Its immutable, read-only filesystem minimizes attack surfaces, and atomic, automated updates keep your system secure and up-to-date without manual intervention." |
| 38 | Red Hat | projectatomic organisation repository listing | source | listing as of 2026-10-07 | 2026-10-07 | https://github.com/orgs/projectatomic/repositories | Generation one's organisation is a graveyard: 69 repositories, the flagship CLI archived, the application spec unmaintained since 2016 | Listing shows atomic (527 stars, updated Oct 9 2020, Public archive), nulecule ("[UNMAINTAINED]", Sep 1 2016, Public archive), atomicapp ("[UNMAINTAINED]", Mar 7 2017, Public archive); total 69 repositories |
| 39 | Red Hat | openshift/enhancements repository | source | 2019-08-21 to 2026-10-06 | 2026-10-07 | https://github.com/openshift/enhancements | Volume and distribution of proposals, derived locally | git log --diff-filter=A on enhancements/: 65 proposals added 2019, 115 in 2020, 111 in 2021, 85 in 2022, 68 in 2023, 62 in 2024, 59 in 2025, 64 in 2026 to 06 October; by area, network 71, ingress 54, installer 51, machine-config 18, rhcos 7 (derived) |
| 40 | Red Hat | rpm-ostree, ostree, bootc commit distribution | source | 2011 to 2026-10-07 | 2026-10-07 | https://github.com/containers/bootc | Where the engineering effort sits moved from the package-aware client to the image client between 2022 and 2024, derived locally | Commits per calendar year (derived): rpm-ostree 1,263 (2021), 1,454 (2022), 530 (2023), 586 (2024), 437 (2025), 81 (2026 to 30 September); bootc 632 (2023), 1,307 (2024), 1,113 (2025), 660 (2026 to 07 October); ostree 915 (2022), 155 (2026) |
| 41 | Red Hat | ostree commit c988ff79, composefs deployment | source | 2023-05-31 | 2026-10-07 | https://github.com/ostreedev/ostree/commit/c988ff79387b5b61a1134c5f0d91010b0a280fd6 | A genuinely read-only root arrived twelve years after the deployment model | Commit subject "deploy: Write a .ostree.cfs composefs image in the deploy dir", 2023-05-31; composefs-touching commits by year (derived): 90 in 2023, 36 in 2024, 6 in 2025, 2 in 2026 |
| 42 | CoreOS Inc | coreos-overlay README | source | archived repository | 2026-10-07 | https://github.com/coreos/coreos-overlay | Generation two was a Gentoo-derived build, which is why nothing in it transferred to the RPM-based successor | "This overlay contains Container Linux specific packages and Gentoo packages that differ from their upstream Gentoo versions." |
Tier mix
| Tier | Count |
|---|---|
| postmortem | 4 |
| source | 18 |
| adr | 14 |
| casestudy | 1 |
| vendor | 5 |
| blog | 0 |
| paper | 0 |
| talk | 0 |
Rows share URLs where one artefact supports more than one claim, as the ledger format requires.
Distinct hosts: 2 (github.com, raw.githubusercontent.com), which is far below this skill's
target of eight and is a consequence of the egress policy rather than of the topic. Anyone
re-running this hunt with open network access should start with the Red Hat and Fedora Magazine
posts, the Flock and DevConf talk recordings, and the RHEL image mode documentation, none of
which could be fetched here.
What the ledger does not contain, and what that costs
- No vendor or engineering blog narrative. The reasoning in this guide comes from design documents written for other engineers, which is better evidence, but it means the product framing and the dates Red Hat itself emphasises are absent.
- No talks or papers. Several of the decisions here were argued out in Fedora Flock and DevConf sessions that issue #2214 refers to; those recordings were unreachable.
- No published post-incident reviews. Red Hat does not publish postmortems for Fedora CoreOS
or RHEL CoreOS releases in any repository reachable here. The four
postmortemrows are issue-tracker reports with reproductions and root causes, and are labelled as such in the page. - No fleet telemetry. Nothing public states how many machines run these systems, what fraction of updates roll back, or how long a bad release takes to withdraw.