Retention, Deletion and the Right to Erasure

A user exercises their right to erasure. The engineer runs a DELETE against the production database. The user's data remains in the previous night's backup, in the analytics warehouse, in three derived tables, in an event log with 90-day retention, in a search index, in a feature store, in log files, and in the training set of a model that has already been deployed. None of that is unusual, and all of it is in scope.

Where copies live

Backups and snapshots. Restoring a backup taken before the deletion reinstates the data. Regulators generally accept that backups are deleted on their normal cycle provided restores are followed by re-application of pending deletions, which requires a durable record of deletion requests that outlives the data.

Immutable event logs. A log is append-only by design, so records cannot be removed in place. Retention expiry eventually removes them, and where retention is long, crypto-shredding is the standard answer.

Table format snapshots. As covered under table formats, deleting a row from the current snapshot leaves it in earlier snapshots that time travel can read. Erasure requires expiring those snapshots and rewriting the underlying files.

Derived and aggregated data. Aggregates over many individuals are usually out of scope, since they are no longer personal data. Derived rows that remain identifiable are in scope, and lineage is what tells you which those are.

Search indexes, caches and feature stores. Each holds its own copy with its own lifecycle and its own deletion path, and each is routinely forgotten.

Models. Training data is not stored in the weights in a retrievable form in general, and memorisation of rare sequences is well documented. Whether a trained model constitutes personal data is genuinely unsettled, and the practical position most organisations take is to exclude erased individuals from subsequent training runs rather than to retrain on request.

Crypto-shredding

The technique that makes erasure tractable on immutable storage: encrypt each subject's data with a per-subject key, store keys in a separate keystore, and delete the key to erase the data. The ciphertext remains and becomes unreadable.

It works well and imposes a real design cost. Every read requires a key lookup, so the keystore is on the critical path and must be highly available. Analytical queries over many subjects need many keys, which is expensive enough that most implementations decrypt into a working set rather than querying encrypted data directly. And the keystore becomes the most sensitive system in the platform, since it is the thing that must be securely and permanently deleted from.

When it breaks

Deletion propagation has no natural mechanism. Nothing automatically pushes a deletion through a derived-data graph. Building it means a deletion event stream that every downstream system subscribes to, with acknowledgement, and treating erasure as a first-class pipeline rather than an operational task.

Regulatory deletion and business retention conflict. Financial records must be kept for years and personal data must be erasable, so the reconciliation is usually pseudonymisation: retain the transaction, remove the link to the identity. That is a schema design decision, made long before the first request, that separates identity from event.

Verification is difficult and expected. Demonstrating that data was erased requires knowing where it was, which requires lineage, which requires the lineage to be complete. Organisations without column-level lineage cannot make the claim honestly, and the audit is where that becomes apparent.

Retention policies are written and not enforced. A documented 90-day retention with no automated expiry means data is kept forever, which is both a compliance exposure and a cost. Enforcement belongs in the platform, expressed as table properties and enforced by a job, not in a policy document.