Datasheets for Datasets
The questions a dataset's documentation has to answer for someone deciding whether to use it, why the composition and collection sections carry the weight, and what happens when a dataset outlives the answers.
A dataset is used by people who did not build it, for purposes its creators did not anticipate, long after anyone remembers how it was made. Datasheets (Gebru et al., 2021, Datasheets for Datasets, CACM 64(12)) borrow the electronics industry's convention that a component ships with a specification of its characteristics and operating conditions, and apply it to data.
The sections and what they are for
Motivation. Why the dataset was created, by whom, and funded by whom. This is context for interpreting every later answer, since a dataset built for one research question carries that question's assumptions in its sampling.
Composition. What the instances are, how many, whether it is a sample and of what larger set, what each instance contains, whether there are labels and how they relate to the instances, whether any data is confidential or sensitive, and whether it identifies subpopulations. This is the section a reuse decision most depends on, because it establishes what the data can support.
Collection. How the data was acquired, over what period, by whom, whether individuals were notified or consented, and whether an ethical review took place. The time period matters more than teams expect: it bounds what the data can represent and predicts when it goes stale.
Preprocessing and labelling. What cleaning, filtering and transformation was applied, whether the raw data was retained, and for labels, who produced them, under what guidelines, with what agreement and what compensation. Label provenance is the most consequential and least documented part of most datasets.
Uses. What the dataset has been used for, what it should not be used for, and what about its composition might create risks in particular uses. As with model cards, the negative statement is the more useful one.
Distribution and maintenance. Licence, how it is distributed, who maintains it, whether it will be updated, and how errors can be reported.
Why it is worth the effort
The value is asymmetric and concentrated. Most of the answers are cheap to write while the knowledge is fresh and impossible to reconstruct later. A dataset whose collection period, consent status and labelling guidelines are recorded remains usable and auditable indefinitely; one without them becomes a file whose properties nobody can establish, which is a liability as regimes requiring provenance take effect.
When it breaks
They are written after the fact. Documentation produced when the dataset is published, by whoever is available, is a reconstruction. Writing during collection is both more accurate and cheaper, and it requires the discipline to treat documentation as part of the work.
Web-scraped datasets resist most of the questions. Consent, notification, individual composition and subpopulation representation are largely unanswerable for a crawl. The honest datasheet says so explicitly rather than omitting the sections, and an omission reads as an absence of concern rather than as an absence of information.
Derived datasets lose the chain. A dataset filtered, deduplicated and reformatted from three sources needs their datasheets plus its own transformation record. Without the chain, the derived dataset's properties are unknowable even when every parent was well documented.
Nobody reads them. Adoption is limited, and a datasheet nobody consults produces no benefit for the effort spent. The counter is to surface the sections that drive decisions, licence, collection period, known limitations, at the point of use in a catalogue rather than in a separate document.
12 flashcards for this concept
Click a card to reveal the answer.