Chunk-Level Deduplication

A team fine-tunes a model and pushes a 4 GB checkpoint. Ninety-five percent of the tensors are byte-identical to the parent checkpoint they started from. A file-addressed store sees a different hash and stores 4 GB. Do that a thousand times a week and the storage bill tracks the number of experiments rather than the amount of new information.

Chunk-level deduplication cuts each file into variable-sized pieces using a rolling hash over the content to choose boundaries, then stores each distinct chunk once. Because boundaries follow the data, inserting or deleting bytes near the start shifts one chunk rather than re-cutting everything after it - the property that fixed-size blocking does not have.

Why it matters

Storage is rarely expensive per gigabyte and frequently expensive in aggregate, and the aggregate in artifact stores is driven by rewrite amplification rather than by new data. The pattern attacks the multiplier directly, and it pays a second dividend on the wire: an upload transfers only the chunks the server does not already hold, so transfer time falls with the same ratio as storage.

It also changes what versioning costs. When a new revision is cheap, teams keep history they would otherwise delete, which is usually the outcome you wanted from retention policy anyway.

Implementation patterns

• Content-defined chunking with a rolling hash, targeting an average chunk size around 64 KB. Smaller averages find more duplication and multiply metadata; larger averages do the reverse.
• A manifest per file listing chunk hashes in order, so reconstruction is a lookup and a concatenation, and the file's own digest is verified on read.
• Tiered duplicate lookup: check the current upload session first, then a local client cache of recently seen chunks, then a bounded global query. A full global check on every chunk makes the index the bottleneck.
• Pack chunks into larger blocks before writing to object storage. Storing 64 KB objects individually turns a storage problem into a request-count problem, since object stores charge per request and perform badly on tiny objects.
• Reference counting with conservative garbage collection. A chunk is deletable only when no manifest points at it, and getting this wrong corrupts many files at once.
• Client-side chunking where possible, so the saving applies to upload bandwidth and not only to stored bytes.

Industry example

The Hugging Face Hub held over 30 PB of models and datasets in Git LFS repositories, where deduplication is file-level: a checkpoint differing in a fraction of its tensors is stored in full. Hugging Face's published measurements for its Xet backend, rolled out from early 2025, reported two versions of GPT-2's model.safetensors falling from 1.2 GB to 645 MB together, a 53% reduction, and the CORD-19 dataset falling from 8.9 GB to 3.52 GB with download time dropping from 51 to 19 minutes. Across fine-tuned models and checkpoints they measured deduplication in the 30–85% range, with PyTorch checkpoints alone representing roughly 200 TB and about 100 TB recoverable at a 50% ratio.

Failure scenarios

• Correlated corruption. A lost or damaged chunk damages every file referencing it, so the blast radius of a storage fault grows with the deduplication ratio. The only honest control is verifying the reassembled digest on read.
• Metadata outgrowing the data it describes. Too small an average chunk size, or many small files, and the chunk index becomes the expensive component.
• Premature garbage collection deleting a chunk whose last reference was added concurrently, which surfaces as a file that downloads and fails its digest check.
• Compressed and encrypted inputs. Chunking a client-side-encrypted or already-compressed artifact finds almost no duplication, because a one-byte change alters every downstream byte.

Trade-offs

You exchange a simple, cheap, stateless storage layer for a cheaper storage layer plus a global index that must be operated, scaled, backed up and kept consistent. Reads gain a reassembly step. Small files get slightly worse. The pattern earns its keep only when the same logical artifact is rewritten many times with small deltas - model checkpoints, dataset revisions, virtual machine images, backups - and not when artifacts are written once and read many times.

When not to use it

For write-once immutable artifacts such as release tarballs or published datasets, file-level deduplication plus a lifecycle rule into a colder storage class captures most of the saving with none of the machinery. For a store below roughly a few hundred terabytes, the index is unlikely to pay for its own operation. And for encrypted-at-rest-by-the-client data, the ratio will be near zero, so measure it on a sample before committing to the design.

Interview question

Q: You run an artifact store whose volume is growing four times faster than the number of distinct artifacts. Would you adopt chunk-level deduplication, and how would you decide before building it?

What a strong answer covers: measuring the achievable ratio on a representative sample first; the distinction between file-level and chunk-level addressing; content-defined boundaries and why fixed blocks fail on insertion; the packing of chunks into blocks to control request counts; correlated-corruption risk and digest verification; the operational cost of a global index; and the conditions under which a lifecycle policy is the cheaper answer.

Quick check

Quiz: Why does content-defined chunking beat fixed-size blocks for deduplication? — Fixed blocks re-align on every insertion or deletion, so a one-byte insert near the start changes every subsequent block. Content-defined boundaries move with the data, so only the affected chunk changes.

Flashcard: What is the failure mode created by a high deduplication ratio? — Correlated corruption: one damaged chunk damages every file that references it, so the blast radius of a storage fault grows with the ratio. Verify the reassembled file digest on read.