A team's registry holds every build artefact ever produced: 400,000 container images, 180 TB, growing 6 TB a month, and the storage bill is now material. Which retention policy should they adopt?
Show the full answer Hide the answer
The deciding property
The value of an artefact is not its age; it is whether anyone can be obliged to reproduce or inspect it. That splits the population sharply. A pull-request build from March has no claimant and can go. An image that ran in production has three possible claimants — a rollback, a security investigation asking whether a vulnerable version ever ran, and an auditor or customer asking what was deployed on a date — and the first is measured in days while the other two are measured in years.
So the policy follows the claimants: a short window for everything, and an indefinite exemption for the set that was deployed or released. That set is small: a team deploying 20 times a day produces perhaps 5,000 production artefacts a year against hundreds of thousands of builds.
Why the others fail
- Keep everything and renegotiate. Buys a year and does not change the curve, and 6 TB a month compounds. It also leaves the registry's own performance degrading, since listing and garbage-collecting hundreds of thousands of tags is not free.
- Last 30 builds per repository. Simple, and it deletes exactly the wrong things. A service that deploys rarely loses its production artefact within a fortnight of a busy week, so the rollback you need is the one that is gone. Count-based retention is age-based retention with an unpredictable clock.
- Only what is running now. Fails the first rollback beyond the current version and every retrospective question. The images you need are precisely the ones no longer running.
What the policy costs, and how to bound it
| Cost | Handling |
|---|---|
| Storage for the indefinite set | Small in practice: ids are cheap and the exempt set is thousands not hundreds of thousands |
| Knowing which artefacts were deployed | Requires deployments to record the digest they shipped, which you need anyway |
| Accidental deletion of something needed | Protect by digest immutability plus a deletion delay and an audit log |
| Layer sharing confusion | Deleting a tag does not free bytes until unreferenced layers are collected, so measure freed space, not deleted tags |
Deduplication changes the arithmetic more than the policy does. Content-addressed storage means shared layers are stored once, and chunk-level deduplication goes further: Hugging Face moved the Hub's large-file storage to Xet after acquiring XetHub in 2024, using content-defined chunking of roughly 64 KB so only changed chunks are stored and transferred. For a registry of near-identical images, the dominant cost is often the few layers that differ per build, which is why measuring bytes by layer before setting a policy is worth an afternoon.
What would flip the decision
- A regulated product where every shipped binary must be retained for a stated period: the exemption becomes the rule, with a documented retention schedule and a legal hold mechanism.
- A product customers install themselves: every published release must remain fetchable for its supported life, so deletion is a support decision rather than a cost decision.
- Artefacts that can be rebuilt deterministically from a pinned source: storage becomes optional for anything you can regenerate bit-for-bit, which is the strongest argument for reproducible builds after security.
When this is over-engineering
A team with 2 TB of artefacts should keep everything and spend the afternoon elsewhere. Retention policy is a response to a bill and to an audit obligation; in the absence of both, the risk of deleting something needed exceeds the saving. The trigger worth watching is not total size but growth rate against the cost of the smallest engineer-week you could spend on it.