advanced 1 min answer

A community platform implements a retention policy deleting user content after a defined period. What happens that nobody planned for?

retentiondeletionreferential-integritybackupsreddit
Show the full answer Hide the answer

The consequences that surface

  • Referential integrity breaks. Deleted content is referenced by replies, quotes, moderation records, aggregates and search indexes. Deleting the row leaves dangling references everywhere, and the systems that read them were never written to handle absence.
  • Aggregates become unreproducible. Historical counts no longer recompute from the retained data, so the reported figure and the recomputed figure permanently disagree. Every restatement now needs an explanation.
  • Derived copies persist. Search indexes, caches, analytics stores, machine learning training sets, data warehouse copies and partner exports. Deletion from the primary store is the easy part; the copies are the actual problem.
  • Backups still contain everything, and restoring one resurrects deleted data.
  • Legal holds conflict with the schedule, and a hold that arrives after deletion is unrecoverable.

What the design must include

  • A deletion propagation mechanism — usually a deletion event consumed by every downstream store, with acknowledgement, so completion is provable rather than assumed.
  • Tombstones rather than hard deletes where references exist, preserving the graph while removing content.
  • Crypto-shredding for backups, since selectively editing backups is not practical.
  • Legal hold evaluated before deletion, as a blocking check.
  • An inventory of every copy, which almost never exists and is the reason deletion programmes overrun.

The framing

Retention is not a delete statement; it is a distributed state change across every system that ever received the data. Designing it late, after the copies proliferated, is what makes it expensive.