Data Lifecycle

Definition

Data has a lifecycle: created, actively used, occasionally accessed, rarely accessed, and eventually deleted. Architectures that model only the first stage accumulate cost and risk indefinitely.

The stages and the decisions

The single most effective intervention is usually the first: not storing it. Sampling telemetry, aggregating rather than retaining raw events, and declining to log full request bodies remove cost and risk permanently rather than managing them.

Industry example

Storage-centric products make tiering economics unusually visible. Dropbox's move to operate its own storage infrastructure was driven by exactly this: at their scale, the cost curve of storage is the business, so the architecture is organised around it — content-addressed blocks that deduplicate identical data across users, erasure coding rather than full replication for durability at lower overhead, and cold data on higher-density lower-cost hardware.

The transferable lessons are two. First, deduplication and erasure coding buy durability far more cheaply than naive replication, and the difference compounds at scale. Second, tiering must be driven by measured access patterns rather than by age alone — a two-year-old file accessed weekly belongs in a hot tier, and an age-based policy would get that wrong.

Deletion, which is the hard part

• Retention floors and ceilings conflict. Tax rules may require seven years; privacy law may require deletion on request. Both apply to the same customer record, and the resolution is field-level rather than record-level.
• Deletion must propagate. Backups, replicas, caches, search indexes, warehouses, logs and third-party processors all hold copies. A deletion that only removes the primary row is not a deletion.
• Immutable stores need crypto-shredding. Encrypt per subject, destroy the key. This must be designed in, because it cannot be retrofitted onto data already written.
• Soft deletes are not deletion for regulatory purposes, and a deleted_at column is frequently mistaken for compliance.

Failure scenarios

• Logs retained forever because nobody set a policy, holding personal data in a system with weak access control.
• Tiering by age alone, moving frequently-read data to cold storage and generating expensive retrieval costs.
• Backups outside the retention policy, so deleted data returns on restore.
• Archive with no tested restore, which is not an archive.

Trade-offs

Aggressive lifecycle management reduces cost and legal exposure and risks deleting something that turns out to be needed. Conservative retention preserves optionality and accumulates both cost and liability — and in a breach, data you did not need is the part that hurts most.

Interview question

"A customer exercises their right to erasure. List every place their data exists in a typical architecture and how you would remove it from each."