A grocery marketplace in the mould of Instacart finds that a lakehouse table reports 9 TB in queries but the object-store prefix holding it is 31 TB, and the gap has widened by about 1 TB a week. Queries are healthy and compaction jobs succeed. Where is the missing 22 TB, and how do you reclaim it safely?

What I would look at first, and why in that order

1. Snapshot count and age. Table formats keep every committed snapshot until it is expired, and each snapshot pins the data files it references. A table compacted hourly with no expiry holds every pre-compaction version forever.
2. Files in the prefix that no snapshot references. Failed or cancelled writers leave data files that were never committed.
3. Write pattern. A streaming sink committing every minute creates 1,440 snapshots a day.

The diagnosis

The "9 TB" is the current snapshot's live data. The other 22 TB splits into two populations, and they need different remedies.

Expired-but-retained history. Compaction rewrites 100 small files into one large file, commits a new snapshot, and the old files remain because time travel to yesterday's snapshot still needs them. With hourly compaction and no expiry, every row has been rewritten many times. Because compaction rewrites rather than deletes, steady storage grows by roughly the rewrite volume per cycle until snapshots are expired.

Orphan files. An executor dies after writing data files but before the commit. The files exist in the prefix and in no snapshot, so no table-level cleanup will ever see them. Spot interruptions and killed jobs produce these constantly.

The misleading signal

Compaction success. It proves the job committed, and says nothing about whether old data was released; compaction is precisely the operation that adds storage until expiry runs. A healthy-looking job is the cause of the symptom.

The fix, in order

1. Set snapshot expiry to the time-travel window the business actually uses (7 days is common) and run it on a schedule. Expiry removes snapshots and deletes files only they referenced.
2. Then remove orphans, with a minimum age (3 days is the default in Iceberg's procedure) so a write in flight is never mistaken for an orphan. Deleting an in-flight file corrupts a commit that has not landed yet.
3. Never delete from the prefix by hand with lifecycle rules keyed on age alone; a lifecycle rule cannot tell a live old file from a dead one.

What it costs

Shortening time travel removes the ability to query or roll back past the window, so agree it with analysts and any audit needs first. Expiry also competes with writers for catalog locks on large tables, so schedule it off the ingest peak.

The alert that would have caught it

Track bucket bytes divided by live-table bytes per table, and page at a ratio above about 2. Add snapshot count per table with a ceiling. Both are cheap and neither depends on a job failing.

Common weak answers

• "Add an object-store lifecycle rule to delete files after 30 days." It deletes live cold partitions along with dead files.
• "Re-run compaction." Compaction is the source of the growth.