Hugging Face acquired XetHub in August 2024 and by 23 May 2025 Xet-backed storage was the default for new users and organisations on the Hub, with byte-level content-defined chunking replacing per-file Git LFS deduplication underneath while Git LFS remained supported. The commands external consumers type did not change. What had to be communicated, which part of a context diagram carries it, and where would copying this be a mistake?
Show the full answer Hide the answer
The situation they were in
Model and dataset files on the Hub are large, and Git LFS deduplicates whole files. A one-line edit to a multi-gigabyte checkpoint is a new file, so it transfers in full. Hugging Face acquired XetHub in August 2024 to replace that layer, and its documentation describes content-defined chunking at roughly 64 KB chunks whose boundaries come from a rolling hash of the contents, grouped into 64 MB blocks stored once in a content-addressed store. Chunk boundaries that follow the data mean an insertion in the middle of a file does not shift every later chunk. Xet-backed repositories became the default for new users and organisations on 23 May 2025, and Git LFS stayed supported for compatibility.
What they chose to hold constant
The consumer-facing contract was a command, not a storage layout. git clone and the Hub client kept working, so the change was invisible to the thing most integrations depend on.
That is only safe if someone has written down which observable properties consumers may rely on. A context diagram is where that lives: the edges from the system boundary out to external consumers are the only place in a diagram set where a promise is distinguishable from an implementation detail. The edge carries the command surface, the URL shape, what the bytes on disk are afterwards — and, separately, what is explicitly not promised: how many bytes cross the wire, what the local cache looks like, which library gives the fast path.
What it cost them
The honest residue is client dependency. Byte-level deduplication needs a client that speaks it; the Python integration ships as a separate hf_xet package introduced with huggingface_hub 0.30.0. A consumer on an older client does not get an error — it falls back to the slower path and sees no message at all. Silent degradation is harder to support than a clean break, because the user reports "downloads feel slow" and nobody can tell which path they took.
So the sentence the announcement has to carry is not "we changed the storage". It is "if your client is older than this version you get the previous path, and here is how to tell which one you got." A performance change with no way for the consumer to observe which branch they are on generates support load indefinitely.
The nine-month gap between acquisition and default is the other cost, and it is the one worth copying: new repositories first, old ones untouched, both backends live at once. Coexistence is what buys the long window.
Where copying this would be a mistake
- If your consumers integrate against the storage layout rather than a command — presigned object URLs, bucket paths, checksum semantics they verify themselves — this is a breaking change whatever your announcement says. The remedy is a versioned endpoint and a dated deprecation window, not better wording.
- If you cannot run both backends simultaneously, the migration is a cutover and the communication problem is completely different: a date, a maintenance window, and a rollback statement.
- If the gain is invisible to consumers, do not announce it as a benefit. Here the gain is measurable by the user — transfer volume on an incremental upload drops by orders of magnitude — so telling them what to re-measure is useful. A change that only improves your own storage bill should be a changelog line.
Common weak answers
- "Publish a migration guide." There is no migration for the consumer to perform, and a guide implies there is.
- "Add it to the architecture diagram." The internal diagrams changed enormously and none of that is the consumer's business. The only edge that needed editing is the external one, and what it needed was the non-promises written down.