A team will publish 30 variants of a 140 GB open-weights model on a hub whose storage deduplicates at chunk level - content-defined chunking with a 64 KB target chunk size and an 8 KB to 128 KB range as the Xet specification documents. Estimate the stored bytes if the variants are full fine-tunes and if they are adapters.

The assumptions, stated

Content-defined chunking splits a file at boundaries chosen by a rolling hash over the content, not at fixed offsets. The documented consequence is the one that matters here: an insertion or deletion only changes the chunks around it, because the boundaries downstream re-synchronise with the content. Dedup then stores one copy of each distinct chunk hash.

A 140 GB model at a 64 KB target chunk size is roughly 2.3 million chunks.

Case one: full fine-tunes

A full fine-tune updates every parameter. Even a tiny change to a float changes its bytes, and at 64 KB a chunk holds on the order of 16,000 four-byte parameters - so essentially every chunk differs from the base model's. Dedup finds nothing.

30 x 140 GB = about 4.2 TB stored, with the dedup ratio close to 1. The only savings are incidental: unchanged tokeniser files, configs and embeddings that were frozen, perhaps a few per cent.

Case two: adapters

A low-rank adapter stores two small matrices per adapted layer, commonly tens to a few hundred megabytes. At 150 MB each: 30 x 150 MB = 4.5 GB, plus one 140 GB base = about 145 GB.

The ratio between the two cases is roughly 29x, and it is a packaging decision rather than a storage one. That is the useful insight: the storage bill for a model hub is set by what contributors upload, not by how cleverly the hub stores it.

Where chunk dedup does earn its keep

• Re-uploads and incremental commits. Fix a config file in a 140 GB repository and only the changed chunks move. The Hub documentation notes the client keeps only chunks the store does not already have.
• Copies between repositories, which become metadata operations over content hashes rather than transfers.
• Format changes that shift bytes without changing content - adding a header, re-ordering metadata, changing a shard boundary. Fixed-size blocks would miss all of these; content-defined boundaries re-sync after the edit.
• Quantised variants built from the same source, where unchanged tensors stay byte-identical.

The decision rule: dedup pays when the bytes repeat, and re-serialising a tensor with new values does not repeat bytes however similar the numbers are. The published trade-off is also worth stating: Hugging Face's writing on moving from chunks to blocks notes that smaller chunks would dedup better and cost more in infrastructure overhead, which is why 64 KB rather than 4 KB.

Which assumption dominates the error

Whether the variants are genuinely full fine-tunes. Many "fine-tunes" freeze most layers, and a frozen layer is byte-identical, so the real ratio sits between the two cases and depends entirely on how much of the network was touched. The chunk size barely matters by comparison: halving it to 32 KB might improve the full-fine-tune case by a per cent or two and doubles the chunk index.

What the number rules in or out

It rules out "storage dedup will handle it" as a plan for a model zoo of full fine-tunes, and it rules in a policy: publish adapters by default, publish merged weights only when a serving stack requires them, and expect a flat 140 GB per merged variant. For the platform operator, it argues for charging or quota-ing by stored unique bytes, since that is the only quantity the contributor can actually influence.

When not to design around dedup

For fewer than a handful of variants the whole analysis is noise: 4.2 TB of object storage costs on the order of a hundred pounds a month, and an adapter-only policy that blocks a release is worth more than that. The failure to watch for is the opposite one - a team that assumes dedup is working, publishes merged weights for months, and discovers at renewal that the hub quota or the egress bill was set by a packaging habit nobody chose deliberately. Measure stored unique bytes per repository once a month; if the ratio is near 1 and the volume is small, do nothing.