Data Versioning and Content Addressing

Code versioning works because source files are small, textual, and meaningfully diffable. Datasets are large, binary, and change by having rows appended rather than lines edited. Every technique that makes Git work fails on a terabyte of Parquet, so data versioning is a different problem that happens to share a vocabulary.

What a version has to identify

A dataset version must uniquely determine the exact bytes used, so that a training run recording it can be repeated. A path is not a version, since the contents at a path change. A date is not a version, since late-arriving data changes what a date's partition holds. A version must be an identifier that cannot describe two different sets of bytes.

Three mechanisms provide this.

Content addressing. Hash each file and record the set of hashes. Two datasets are identical if their hash sets match, deduplication is automatic since identical files are stored once, and integrity is verifiable. Tools in this family store the hashes in Git alongside the code and the bytes in object storage, so the repository stays small while the reference is exact.

Table snapshots. A table format's snapshot ID already identifies the exact set of files constituting the table at a moment. Recording that ID in run metadata is the cheapest correct answer available when the data lives in Iceberg or Delta, and it costs nothing extra because the metadata exists anyway.

Immutable partitions. Write each batch to a path that is never modified, and define a version as a set of partition identifiers. Simple, requires discipline that nothing rewrites history, and provides no integrity checking.

Why the query matters as much as the bytes

A version that identifies a table snapshot does not identify the training set, because the training set is the result of a query: filters, a train-test split, a sampling rate, a deduplication step. Two runs over the same snapshot with different filters used different data.

So the recorded version has to be the snapshot plus the transformation, which in practice means the query or the preprocessing code version. This is why data versioning and code versioning cannot be separated: the dataset is a function of both, and recording either alone leaves the other free to vary.

When it breaks

Hashing a terabyte is not free. Content addressing requires reading everything to hash it, which for a large corpus is a substantial job. Incremental hashing of only changed files makes it tractable and requires knowing which files changed, which brings back a dependence on the storage layer's own change tracking.

Lineage across derived datasets multiplies versions. A model trained on a dataset derived from three sources needs all four versions, and any of them changing changes the result. Without automated propagation this is recorded by hand and is recorded incompletely.

Erasure conflicts with immutability. A content-addressed store that keeps every version forever cannot honour a deletion request without breaking the addressing of every version containing that record. Reconciling the two means either accepting that old versions become unreproducible after erasure, or excluding personal data from the versioned corpus entirely.

Storage grows faster than intuition suggests. Deduplication helps when files are identical and does nothing when a row is appended to a large file, since the file's hash changes and both copies are stored. Datasets that grow by rewriting large files, rather than by adding new ones, defeat deduplication almost completely, which is an argument for immutable partitioned writes at the storage layer.