intermediate 3 min answer

Litigation requires preserving every record relating to one counterparty, overriding a retention policy that deletes after seven years. The data sits in a 400 TB object-store data lake, roughly 60,000 Parquet files, and the relevant rows are perhaps 3% of them spread across most files. Estimate the cost of implementing the hold, and say what you would do.

legal-holdobject-storagetable-formatsretentionestimation
Show the full answer Hide the answer

The assumptions, stated

  • 400 TB across about 60,000 Parquet files, so files average roughly 7 GB.
  • The relevant rows are about 3% of records and appear in most files, because the lake is partitioned by date rather than by counterparty.
  • Retention is enforced by a lifecycle policy that deletes objects older than seven years.
  • Object storage at roughly $20 to $25 per TB per month for standard tiers, and under $5 per TB per month for archive tiers, as a planning figure rather than a quote.

The arithmetic, three ways

Option A: hold everything. Suspend the lifecycle rule and keep all 400 TB. Cost is about 400 × \(22 ≈ **\)8,800 a month**, around $105,000 a year, continuing for the life of the matter, which can be several years. Implementation is one configuration change and an afternoon.

Option B: extract the relevant rows into a preserved dataset. 3% of 400 TB is 12 TB, about $264 a month. The work is a scan of the full lake to find the rows (one full read of 400 TB plus compute), a defensible extraction process, and documentation of the query used. Call it two to four weeks of engineering plus a few thousand dollars of compute.

Option C: row-level hold in place. Not available in a plain object store. Rewriting files to isolate the 3% means rewriting most of the 60,000 files, which is a full-lake rewrite — the most expensive option and the one most likely to be challenged, because rewriting the files that contain the evidence is exactly what a legal hold is supposed to prevent.

Which assumption dominates the error

The duration of the matter, not the volume. Option A at a year is $105,000 and at five years is over half a million, which is where the arithmetic turns. The second-largest factor is whether the extraction is defensible: if the process cannot be explained and reproduced, Option B's saving evaporates into a dispute about spoliation.

What the numbers rule in or out

  • Start with Option A immediately, within hours. Suspending the lifecycle rule is reversible, cheap in the short term, and the only action that guarantees nothing is lost while you think. Deleting data after a hold is notified is the one unrecoverable mistake, and it costs vastly more than a year of storage.
  • Then move to Option B if the matter will run, with the extraction documented, hashed and witnessed, and the original still held until the extracted set is verified.
  • Use object-lock or immutability features on the preserved set rather than relying on a policy, so no later automation can remove it. A policy that can be changed by a pipeline is not a hold.
  • Rule out Option C unless the lake uses a table format with time travel and snapshot retention (Iceberg, Delta, Hudi), in which case pinning a snapshot achieves a point-in-time hold without rewriting data. If you are designing a lake today, this is the feature that makes legal hold tractable, and it costs you snapshot storage you would otherwise expire.
  • Suspend the deletion path, not just the schedule. Holds fail through the side door: a compaction job that rewrites files, a cost-optimisation that re-tiers to a lifecycle with its own expiry, or a backup rotation that ages out independently.

When this is the wrong question

For a matter where the regulator or court has specified preservation of a defined scope, the cost is not a decision variable and Option A is correct regardless of price. The estimate matters when the organisation has choices, and the engineering answer is to make those choices available in advance: partition by the dimension holds are likely to follow, adopt a table format with snapshots, and keep the deletion path auditable. All three are cheap at design time and unavailable afterwards.