Retention & Purge
Deleting from an append-only estate, and proving the deletion happened.
6 to work through
-
intermediate
A subject erasure request arrives for one customer. Their rows sit in a lakehouse table of 4 billion rows in 1,100 partitions, with 30 days of snapshot retention, plus a search index and two downstream marts. Roughly what does honouring it cost, and which assumption dominates?
3 min answer -
advanced
A community platform implements a retention policy deleting user content after a defined period. What happens that nobody planned for?
1 min answer -
advanced
An erasure request arrives for a customer whose data has propagated into a warehouse, a lake, search indexes, caches, backups and trained models. How is this handled, and what should have been designed in advance?
2 min answer -
advanced
An insurance platform must delete data on request while retaining records for regulatory periods. What design satisfies both?
2 min answer -
advanced Multiple choice
On 5 April 2022 Atlassian ran a maintenance script to deactivate a legacy app. The script carried both a mark-for-deletion mode used in daily operations and a permanent-delete mode kept for compliance erasure. It ran in the wrong mode against site identifiers rather than app identifiers and destroyed 883 sites belonging to about 775 customers. Restoring them took up to two weeks. Which design change most directly prevents a repeat?
2 min answer -
advanced
Your platform is built on an append-only event log with a lakehouse behind it. Legal asks how you will satisfy erasure requests within 30 days. What is your answer?
2 min answer
4 terms in this topic
Deletion Propagation
The mechanism that carries a deletion from the primary store to every derived copy - which is where retention and erasure obligations are actually me…
practiceDeletion Verification
Proving that a deletion actually removed the data everywhere it existed, which is a harder problem than issuing the delete.
conceptErasure Amplification
The ratio between the bytes rewritten to honour a deletion request and the bytes actually belonging to the subject, which decides whether on-demand e…
practiceRetention and Purge
Deleting data when its lawful or business purpose has ended, and being able to prove that it happened.
Neighbouring topics
Data Governance & Semantics
General material on ownership, meaning, quality and control of data at enterprise scale.
Data Mesh
Domain ownership, data as a product, self-serve platform, and federated governance.
Data Products
A dataset with an owner, an interface, an SLO, and consumers who can rely on it.
Data Contracts
Producers committing to schema, semantics and freshness, and breaking builds when they do not.
Data Catalog
Discovery, ownership and technical metadata, and why catalogues go stale.
Business Glossary
Agreeing what a term means before arguing about which number is right.
Semantic Layer
Metric definitions held once and served to every tool that asks.
Master Data Management
One authoritative record for a customer or product across systems that each have their own.
Reference Data
Code lists, hierarchies and currencies — small, shared, and quietly load-bearing.
Data Quality Dimensions
Completeness, accuracy, timeliness, consistency, validity and uniqueness as testable claims.
Data Observability
Freshness, volume, schema and distribution monitoring for pipelines that fail silently.
Data Stewardship
The operating model that makes ownership a role rather than a slide.
Data Access Models
Role, attribute and purpose-based access over analytical data, and how they compose.
Row & Column-Level Security
Restricting slices of a table rather than the whole table, and where it is enforced.
Tokenisation & Masking
Dynamic masking, deterministic tokens, and preserving joinability without exposure.
Data Sharing & Clean Rooms
Collaborating on data neither party may hand over, with computation as the interface.
Sensitivity Labelling
Propagating a classification through joins and derived tables so controls follow the data.
BI Governance
Dashboard sprawl, certified reports, and the number the board is allowed to see.
Self-Service vs Governed
Letting analysts move fast without four teams reporting four different revenues.