The Hugging Face Hub held over 30 PB of models and datasets in Git LFS repositories where every new checkpoint version was stored in full. Sequence the move to chunk-level deduplicated storage under live traffic, and say where the saving actually comes from.

Where the saving comes from

Git LFS deduplicates at file granularity: two objects are shared only if their hashes match exactly. A fine-tuned checkpoint that differs from its parent in a fraction of its tensors is a completely new multi-gigabyte object, so the Hub paid full price for near-identical bytes.

Xet, the storage backend Hugging Face began rolling out in early 2025, deduplicates at chunk granularity using content-defined chunking: a rolling hash over the file contents picks boundaries at roughly 64 KB average, so an insertion or deletion shifts one chunk rather than re-cutting the whole file. Chunks are hashed and checked against the upload session, a local cache, and then global storage.

Hugging Face's own measurements published with the design: two versions of GPT-2's model.safetensors fell from 1.2 GB to 645 MB together, a 53% reduction. The CORD-19 dataset fell from 8.9 GB to 3.52 GB and download time from 51 to 19 minutes. Across fine-tuned models and checkpoints they measured deduplication ratios in the 30–85% range, with PyTorch checkpoints alone representing roughly 200 TB and about 100 TB of that recoverable at a 50% ratio.

The migration sequence

1. Run the new backend behind the old protocol. Clients that speak only Git LFS keep working; the gateway chunks on write and reassembles on read. Nothing about the repository's public identity changes.
2. Migrate read-heavy, high-duplication repositories first — checkpoint families and dataset revisions — because that is where the ratio is measurable and the win pays for the transfer.
3. Ship the chunk-aware client and let it opt in per repository, so upload savings arrive only where a client can compute chunk boundaries locally.
4. Backfill the long tail at a rate the object store's request budget tolerates.
5. Keep the old path readable indefinitely. A public artifact store cannot break a URL that a training script pins.

Where data can diverge, and how you would know

Chunk-level storage means a corrupted or lost chunk damages every file that references it, so the blast radius of a storage fault grows with the deduplication ratio. The controls are verification of the reassembled file digest against the original object hash on read, and a reference count that never allows a chunk to be garbage collected while any manifest points at it. The signal that something is wrong is a digest mismatch rate above zero, not a storage error.

What it costs

Metadata. Every file becomes a manifest of chunk references, and the chunk index is a global structure that must survive and be queried on every upload. You trade a cheap dumb storage layer for a cheaper storage layer plus a database you now operate. Small files get worse, not better: chunking a 4 KB config adds overhead and saves nothing.

When not to do this

If your artifacts are write-once and rarely versioned — release binaries, immutable datasets — file-level dedup plus a lifecycle policy to a colder class captures most of the saving with none of the machinery. The chunk design pays off specifically when the same logical object is rewritten many times with small deltas, which is what a model checkpoint is and a release tarball is not.