Erasure Amplification

One customer asks to be deleted. Their data is perhaps 200 kilobytes. Honouring the request rewrites 200 gigabytes.

The multiplier comes from immutability and from scatter. Analytical storage is made of immutable files, so removing a row means rewriting the whole file that contains it. A customer who transacted over three years has rows in hundreds of date partitions, and each partition contributes at least one file to rewrite. The cost is set by files touched, not by rows deleted, and the two numbers can differ by six orders of magnitude.

Why it matters

It decides the shape of the compliance design rather than its details. A platform that assumes erasure is a DELETE will meet its first busy week and discover the table is being rewritten continuously and never finishes. The right question is not "how do we delete a row" but "what is the longest deletion cycle our policy and our regulator accept" — and asking it that way makes a hard engineering problem into a scheduling one.

It also changes what may honestly be said to a regulator. Rewriting files erases nothing while an unexpired snapshot still references the old ones, so the true completion time is the rewrite plus the retention window.

Implementation patterns

• Batch into purge cycles. A hundred requests in one pass cost barely more than one, because the same files are rewritten either way. A weekly cycle sits comfortably inside a one-month statutory response window.
• Keep a tombstone registry recording each request, when it was received and which cycle satisfied it. This is the evidence, and it must survive the deletion it describes.
• Localise the subject in the partition space where the data model allows it. Data partitioned so that a subject's rows cluster turns hundreds of file rewrites into a handful — though this usually conflicts with the date partitioning that analytics wants, which is the real tension.
• Crypto-shred where volume forces it: encrypt per subject with a key in a key store, and erase by destroying the key. Deletion becomes constant time regardless of data volume.
• Alert on the age of the oldest outstanding request, never on purge-job success. The silent failure is a cycle that overruns its window.

Industry example

Statutory erasure rights have been operative in Europe since 2018, and the two designs above are now the documented industry answers. The split is consistent: organisations with low request volume rewrite on a schedule and keep a registry, while organisations holding subject data at high volume across many derived copies adopt key-based erasure because the rewrite cost does not converge. Neither is a compliance choice; both satisfy the obligation. It is a volume decision.

Failure scenarios

• On-demand deletion accepted as a design, which works in the pilot at three requests a month and collapses at five hundred a week.
• Snapshot retention ignored, so a compliance report states completion on the rewrite date while the data remains readable through time travel for another 30 days.
• A lost encryption key for a subject whose data was still needed, which is unrecoverable data loss rather than a delay — the inverse failure of crypto-shredding and the reason key custody becomes a correctness dependency.
• Derived copies missed because the purge was scoped from incomplete lineage, leaving identifiable rows in a mart nobody knew consumed the source.
• Search indexes and caches forgotten, which hold the same records with different retention and no purge path at all.

Trade-offs

When not to use it

A dataset under a few hundred gigabytes with a handful of requests a year should simply be rewritten, and building any of this machinery for it adds a permanent operational dependency to avoid a cost nobody is paying.

The obligation is also narrower than teams assume. Aggregates that cannot identify an individual generally fall outside erasure scope, so the most effective intervention is usually not faster deletion but holding identifiable data in fewer places — a modelling decision taken years before the first request arrives. The platform that ingested a customer identifier into eleven marts has an erasure problem; the one that resolved it to a surrogate at the boundary largely does not.

Interview question

Q: Your lakehouse holds 4 billion rows across 1100 partitions with 30 days of snapshot retention, plus a search index and two marts. A subject erasure request arrives. Size the work, then tell me what you would change before the volume of requests grows tenfold.

What a strong answer covers: files touched rather than rows deleted as the cost driver, with a worked estimate · batching as the design lever, and framing the question as the acceptable cycle length · snapshot retention meaning the honest completion date is later than the rewrite · a tombstone registry as the evidence · crypto-shredding as the volume answer with its key-custody and query-performance costs · the search index and caches as the parts most often forgotten · and reducing the number of identifiable copies as the intervention with the best return.

Quick check

Quiz: Why can deleting one customer's 200 KB rewrite 200 GB? Because immutable files must be rewritten whole and the subject's rows are scattered across hundreds of partitions, so cost follows files touched rather than rows deleted.

Flashcard: What should a purge pipeline alert on? The age of the oldest outstanding request, not purge-job success — the failure is a cycle that quietly overruns its window.