Snapshot Expiry

A lakehouse table that supports time travel keeps the file set of every past commit. That is what makes AS OF queries and rollbacks possible, and it is also why a table whose rows are updated daily can hold ten times its logical size on disk within a year.

Nothing reclaims that space automatically. A compaction job that rewrites 500 GB into larger files does not delete the 500 GB it replaced, because an older snapshot still points at it. The storage bill grows, the metadata grows with it, and query planning slows as manifest lists lengthen.

Snapshot expiry is the maintenance job that drops snapshots older than a retention window and then deletes the data files no surviving snapshot references. It is the single most commonly missing job on a new lakehouse.

Why it matters

Storage is the visible cost and rarely the largest one. Planning time grows with the number of snapshots and manifests, so a table with 200,000 retained snapshots can spend seconds in planning before it reads a byte, and that cost is paid by every query, including the cheap ones.

The second reason is legal. If an erasure obligation requires that a customer's rows are gone, a row deleted from the current snapshot is still present in every retained older snapshot. Without an expiry policy shorter than the obligation's deadline, the deletion did not happen in any sense a regulator would accept.

Implementation patterns

• Set retention from a stated requirement, not a feeling: the longest rollback window operations actually uses, usually 3–7 days, or the longest audit query the business runs.
• Run expiry on a schedule and alert on its failure. It is a maintenance job, so it fails quietly and nobody notices for months.
• Pair it with orphan-file cleanup. Failed writes leave data files that no snapshot ever referenced; expiry does not touch them because it only follows snapshot references.
• Keep one long-lived tagged snapshot where a business requires a fixed reporting baseline, rather than lengthening retention for the whole table.
• Sequence it after compaction, or compaction's replaced files stay referenced and the rewrite doubles storage instead of reducing it.

Industry example

The maintenance burden is why managed lakehouse services began shipping automatic table optimisation from around 2022: compaction, clustering and expiry run as a service because the self-managed version of all three is where teams reliably underinvest. Datadog's published description of Husky's compaction makes the same point from the other side — its compactors exist to keep fragment counts bounded, and bounded fragment counts are only bounded if the replaced fragments eventually go away.

Failure scenarios

• Storage grows steadily on a table whose row count is flat. The classic signature of updates plus no expiry.
• Expiry with a retention window shorter than a running job. A long Spark read holding a snapshot that is then expired fails mid-query with missing files.
• Deleted data resurfacing through a time-travel query after an erasure request was reported as complete.
• Cross-table references: a downstream table pinned to a snapshot id that expiry removes, breaking a reproducible pipeline that was correct yesterday.
• Expiry and compaction scheduled to overlap, so compaction's output is written while its input is being deleted.

Trade-offs

When not to use it

On an append-only table that is never updated or deleted from, expiry reclaims almost nothing, because old snapshots reference the same files the current one does. Running it is harmless and the payoff is metadata pruning only. Equally, do not expire aggressively on a table under active migration: during a parallel run, the ability to compare today against last Tuesday is worth more than the storage.

Interview question

Q: A lakehouse table holding 2 TB of logical data occupies 14 TB in object storage, and its row count has been flat for six months. Walk me through what you would check, in order, and what you would change.

What a strong answer covers: update-heavy writes plus retained snapshots as the first hypothesis; checking snapshot count and total referenced versus unreferenced bytes before changing anything; distinguishing retained snapshots from orphan files, which need a different job; setting retention from the real rollback requirement; sequencing expiry after compaction; and the erasure-obligation angle, which usually turns a cost conversation into a compliance one.

Quick check

Quiz: Why can a table's storage triple while its row count stays flat? — Because updates write new files while old snapshots keep the replaced ones alive, and nothing deletes them until snapshots expire.

Flashcard: Two jobs are needed to reclaim lakehouse storage. Which, and what does each miss on its own? — Snapshot expiry removes files no live snapshot references; orphan cleanup removes files no snapshot ever referenced. Expiry alone leaves failed-write debris; orphan cleanup alone leaves every superseded version.