Dataset Deprecation Policy

A team publishes a dataset. Two years later it holds 40 columns, six of which nobody has read since 2024, and a source system change means one of those six must now be maintained or dropped. The team does not know who reads it, so the answer is to maintain it.

Repeat that a few dozen times and the estate is carrying most of its history permanently, with a maintenance bill that grows and never falls. The missing artefact is not tooling, it is a policy: the rule that says what it takes to remove something.

A deprecation policy states the notice period, how consumers are identified, what a consumer is entitled to during the notice, and what happens at the end if they have not moved.

Why it matters

Deprecation is the only mechanism by which a data estate gets smaller. Without it, every schema is append-only in practice, and the cost of every change grows with the accumulated surface.

It also determines whether a data product's contract means anything. A product promises stability; stability without a defined way to change is a promise to freeze, and teams that have promised to freeze stop publishing anything they might later regret.

Implementation patterns

• Publish usage telemetry from day one: who queried which columns and when. This is the one piece that cannot be added retroactively, and it costs almost nothing at the start.
• Set a notice period by tier: 90 days for a widely used product, 30 for an internal one. Write it before the first consumer, when it is unarguable.
• Deprecate in place first. Mark the column deprecated in the catalogue, keep serving it, and alert its remaining readers by name. Most of the migration happens here.
• Then break it loudly and reversibly: null the column or fail the query for a scheduled window — an hour, then a day — before removal, so the last unknown consumer surfaces while the change can still be undone.
• Version the interface rather than the table, so a consumer can pin to v1 while v2 ships.

Industry example

The discipline is inherited from public API management, where deprecation policies with stated sunset periods have been standard practice for years and the HTTP Sunset header was standardised in RFC 8594 in 2019. The data estate is usually a decade behind on this, for one structural reason: API consumers are identifiable by credential, while a dataset's consumers are whoever wrote a query, so the telemetry that makes deprecation possible has to be built deliberately rather than falling out of the request path.

Failure scenarios

• No usage telemetry, so the consumer list is a guess and removal is never approved.
• A notice period with no enforcement, so consumers learn that deadlines are advisory and the next deprecation takes twice as long.
• Removal without a trial break, so an unknown consumer's pipeline fails on a Sunday with nobody available who knows why.
• Deprecating the physical table while consumers depend on the view, which breaks people who did everything right.
• A deprecated column still being written, so a downstream team keeps building on it because it demonstrably works.

Trade-offs

Short notice periods keep the estate small and push migration cost onto consumers, sometimes at a bad time for them. Long ones are kind and can mean a team maintains two versions for a year. The cost lands on whoever has less slack, and a policy written in advance is how that decision gets made once instead of being renegotiated per column by whoever argues hardest.

When not to use it

A dataset with one consumer in the same team does not need a policy; it needs a conversation. The ceremony pays for itself when consumers are people the producer does not sit with. It is also the wrong instrument for an emergency: if a column must go for a security or legal reason, that is an incident path with its own rules, and pretending it is a deprecation wastes the notice period you do not have.

Interview question

Q: You own a published dataset with 11 known consumers and no usage telemetry. A source change forces you to drop two columns within 60 days. Walk me through your plan.

What a strong answer covers: instrumenting reads immediately even though it is late, because 30 days of telemetry beats a survey; announcing with a named owner per consumer rather than a broadcast; deprecating in place and alerting remaining readers; a scheduled short break before removal to surface unknown consumers while it is reversible; offering a compatibility view for the two columns where feasible; and being explicit that the 60 days is driven by an upstream constraint, not chosen, because consumers respond differently to a deadline someone else set.

Quick check

Quiz: What single measurement makes deprecation possible? — Per-consumer, per-column usage telemetry, collected from the day the dataset is published.

Flashcard: What is a trial break and why does it come before removal? — A short scheduled failure of the deprecated field, which surfaces unknown consumers while the change can still be reverted.