Reliability & Consequence 8 September 2026 7 min read 1,648 words

Custody is not provenance

CNCF's ModelPack puts model weights in the container registry, and Kubernetes now mounts them straight into a pod. The format is sound. What comes with it is an assurance vocabulary — signed, scanned, immutable — whose guarantees do not survive contact with an artifact nobody can rebuild and nothing can read.

The argument

Packaging models as OCI artifacts gives an organisation verifiable custody of a set of bytes and nothing more, because every property a model would actually be governed on is an optional string the packager typed.

Somewhere in your registry, if you have started down this road, there is an object that looks exactly like a container image and is not one. It has a manifest. Its layers carry sha256 digests. Its config is referenced by digest, the way every image you have ever shipped is. In the example manifest the specification itself publishes, one layer is 5,018,536,960 bytes long. Your admission controller will let it through, and it will be right to, by every rule you have written.

The object is a model. The CNCF's ModelPack specification defines how to put one there: an OCI image manifest with an artifactType of application/vnd.cncf.model.manifest.v1+json, weights carried as layers under media types like application/vnd.cncf.model.weight.v1.raw, with parallel types for tokenizer configuration, documentation, code and even datasets. The implementers' note is practical and revealing — package weights without compression, because "model weights files are typically incompressible" and decompression is pure overhead. And since Kubernetes graduated the ImageVolume feature to stable in v1.36, having carried it from alpha in v1.31 and beta in v1.34, the last mile is closed too. The registry object becomes a read-only volume in a pod. No init container, no sidecar that curls from a bucket, no shared filesystem with a naming convention nobody wrote down.

This is a genuinely good piece of engineering solving a genuinely embarrassing problem. It is also the moment to be precise about what it establishes, because the answer is narrower than the packaging implies, and the gap is where the next few years of governance theatre will live.

The specification is candid, in a sentence most people will skim past. It "only contains part of the model metadata, and handles model artifacts as opaque binaries." Opaque. Everything the platform will ever know about the model, other than its size and its hash, is in a JSON config document — and that document is worth reading field by field.

Two things in it are required. modelfs.type, which must be the string "layers". And diffIds, the array of layer content hashes. That is the mandatory content of a model's identity document: a declaration that it is made of layers, and the hashes of those layers.

Everything that describes the model is optional. family, version, vendor, licenses as SPDX expressions, datasetsURL — "a list of links or references to datasets that the model was trained upon". architecture, format, paramSize, precision, quantization. And a capabilities object with inputTypes, outputTypes, languages, a knowledgeCutoff timestamp, and three booleans: reasoning, toolUsage, reward. Whether the model can perform reasoning tasks is a boolean, OPTIONAL, supplied by whoever ran the packaging tool.

None of these are derived. Nothing in the pipeline computes them, checks them against the bytes, or fails when they are absent or wrong. They are what the packager typed. The registry will store them with perfect fidelity and verify, to a cryptographic standard, that they have not changed since — which is a real guarantee about a set of assertions, and no guarantee at all about a model.

The specification is honest enough to have a field for this. Among the layer annotation keys is org.cncf.model.file.mediatype.untested, which "signals that the model packager has not verified the media type classification and the type is inferred or assumed based on some heuristics." Read one way, that is admirable: a standard that ships a flag for its own uncertainty. Read another, it is the shape of the whole config in miniature. There is a defined way to record that nobody checked. There is no defined way to record who is willing to be wrong about it.

Compare this with the artifact whose tooling is being borrowed. For a container image, the claims that matter are mostly derivable from the artifact itself. A scanner opens the layers, finds the package database, enumerates what is installed, produces an SBOM and matches it against advisories. It does not need the image author to be honest, because it can read. Provenance attestations anchor to a build that can be re-run from source. Even the layer model earns its keep: a base layer is shared across a thousand images and pulled once.

Almost none of that transfers. An opaque, uncompressed, incompressible weights blob has no package database and no readable structure. Layer sharing does little for you when a fine-tune changes every byte and produces a wholly new digest. And the specification's own Reproducibility section — files added in lexicographical order, file metadata "set to known, constant values rather than the current values on disk", platform-specific metadata omitted — makes the tar reproducible, not the model. You can rebuild the wrapper deterministically. You cannot rebuild what is inside it, and neither can anyone else.

Here the strongest objection arrives, and it deserves to be put properly rather than waved at.

First: images were never better. The OCI annotations specification says outright that the property "contains arbitrary metadata", that consumers "MUST NOT generate an error if they encounter an unknown annotation key", and it defines org.opencontainers.image.licenses and org.opencontainers.image.source as exactly the sort of free-text field a Dockerfile author fills in unverified. The licence string on your base image is no more attested than ModelPack's. Second, and heavier: look at what this replaces. The status quo is a script that pulls from object storage over an unauthenticated path, a filename convention, and a shared mount that four teams write to. Against that, content-addressed immutability, registry RBAC, mirroring, pull-through caching, air-gapped transport and a digest reported in the container status — Kubernetes v1.36 also added ImageVolumeWithDigest for precisely that — are not marginal. They are the first time model distribution has had any operational hygiene at all. And the specification calls itself current work, with a runtime specification sitting in a later roadmap phase. Faulting a v1 packaging format for not having solved provenance is unfair on its face.

All of that is correct, and none of it is the error. The error is not in the format. It is in what an organisation will infer from the location.

Assurance in most enterprises is procedural, and procedures attach to places. Once a model lives in the registry, it inherits the pipeline built around the registry, and every gate in that pipeline will pass. Signed by an approved key: yes. Pulled from an approved registry: yes. Scanned: yes — and this is the one to sit with. For an image, a clean scan is weak evidence, but it is evidence; something opened the artifact and understood part of it. For a weights layer, the scanner has nothing it can parse. The result is not a failure and not an exemption. It is a green check with no content behind it, and a green check is not neutral — it retires the question. Nobody re-asks what the pipeline has already answered.

Even the projects furthest along concede where the boundary sits. Sigstore's model-transparency work signs a list of file-path-and-digest pairs, and its own description of what verification buys you is that "the model hasn't been tampered with after training." After training. The signature carries custody forward from the trainer; it is silent on everything upstream of that moment, which is where every question an architect actually has — what data, what licence, what was optimised for — happens to live. The project notes an intention to use its predicates "to store (and therefor sign) model card information in the future". Signing the claims is future work in the effort that has thought hardest about it. ModelPack's roadmap, four phases from packaging to runtime specification, does not mention signing at all.

So the discipline worth adopting is to keep two questions apart that the registry quietly merges. Are these the bytes we approved? The registry answers that beautifully, better than anything the field has had before, and that answer is worth the migration on its own. Should these bytes have been approved? Nothing in the manifest touches it. That answer has to be produced by a person or a process willing to be accountable for it, and attached to the digest as an attestation someone signed — not typed into an optional string that travels beside the weights looking exactly as authoritative as the hash next to it.

The distinction stops being philosophical the first time something goes wrong. A training set turns out to have been poisoned, or a licence turns out not to have permitted what you did with it, and the questions arrive in a fixed order: which model artifacts are affected, and where are they running right now. The second question this stack answers well — a digest in the manifest, a digest in the container status, mounted volumes you can enumerate across every cluster you own. The first it cannot answer at all, because the link between a digest and the training run that produced it exists nowhere in the format. You will end up asking the team that packaged it, from memory, months later. That is exactly where the industry stood before any of this, with better tooling wrapped around it.

The specification opens with a story about four ages of infrastructure: machine, virtual machine, container, and now model. It is a good frame, and it carries a warning the authors did not intend. Each age inherits the previous age's tooling, and with the tooling it inherits an idea of what a passing check means. The container age spent a decade learning to distrust artifacts it could not rebuild. The model age begins by placing an artifact nobody can rebuild into the slot reserved for the ones we could.

The digest will match. It will go on matching, every pull, for years. That is the part worth worrying about.

What this is argued from

Reporting and primary material the piece rests on, dated at the time of writing. The interpretation is mine; the facts belong to these.

  1. CNCF ModelPack Specification Standard CNCF ModelPack · 2026-09-08
  2. Model Format Specification CNCF ModelPack · 2026-09-08
  3. Model Artifact Configuration CNCF ModelPack · 2026-09-08
  4. Annotations CNCF ModelPack · 2026-09-08
  5. ModelPack Roadmap CNCF ModelPack · 2026-09-08
  6. KEP-4639, OCI images as VolumeSource Kubernetes · 2026-09-08
  7. Kubernetes CHANGELOG-1.36 Kubernetes · 2026-09-08
  8. OCI Image Format Specification, Annotations Open Container Initiative · 2026-09-08
  9. Model Transparency Sigstore · 2026-09-08

Editorials on this site are written to be argued with. If you think the reading is wrong, it probably is in some particular way, and that is the useful part.

model packagingsupply chainoci registryprovenancekubernetes