Compaction

Streaming and micro-batch ingestion write frequently, and frequent writes produce many small files. Over weeks a table accumulates thousands or millions of them, and the degradation is gradual, silent and severe:

• Per-file read overhead. Opening a thousand small files is dramatically slower than opening ten large ones, and object storage per-request pricing makes it expensive as well as slow.
• Metadata growth, since every file is tracked in the table's metadata, which itself becomes a scaling problem.
• Weak statistics and poor compression, both of which work better over larger row groups — which degrades pruning, and pruning is the main performance mechanism the table format provides.

Compaction is therefore a required background process rather than optional maintenance, and its absence is the most common and least dramatic lakehouse failure.

Why it matters

The failure has no incident. Nothing breaks; queries get slower and more expensive by a few percent a week, which is below the threshold at which anyone investigates. A year later the table is an order of magnitude more expensive to query than it needs to be, and the cause is invisible without looking at file counts.

It also compounds with the layout problem: small files have wide statistics, so pruning fails, so more data is scanned, so compute is added — and the root cause is a maintenance job nobody scheduled.

Implementation patterns

• Trigger on file count or average file size crossing a threshold, rather than purely on a schedule, so busy tables are compacted more often than quiet ones.
• Target file sizes in the hundreds of megabytes — large enough to amortise per-file overhead, small enough that pruning granularity is useful.
• Sort or cluster during compaction on the columns queries filter by, so the job improves pruning rather than merely reducing file count. This is where most of the value is, and compaction that only merges files leaves it on the table.
• Treat compaction as a writer subject to the format's concurrency control, and serialise it against other operations that rewrite files — backfills, corrections — since those conflict with everything and retrying a large rewrite is expensive.
• Expire old snapshots and remove orphaned files, or storage grows without bound and time travel retains every version forever.
• Partition by writer where possible, so streaming ingestion and compaction touch disjoint partitions and never conflict.
• Monitor file count, average file size and bytes scanned per query, since these are the leading indicators and nothing else surfaces the problem.
• Budget the compute, because compaction is a real ongoing cost that must be weighed against the query savings — which it reliably wins.

Industry example

Every production lakehouse deployment confronts this, and the table formats provide explicit operations for it — OPTIMIZE and its equivalents — precisely because the problem is universal rather than incidental. Published guidance from the major platforms treats compaction, snapshot expiry and orphan-file cleanup as standard operational responsibilities, comparable to vacuuming in a relational database.

The organisational failure is also universal: a team adopts a lakehouse for its transactional guarantees and open format, does not schedule maintenance, and concludes eighteen months later that the platform is slow — attributing to the technology what is actually an unperformed operation.

Failure scenarios

• No compaction scheduled at all, producing gradual silent degradation.
• Compaction that merges without sorting, capturing a fraction of the available benefit.
• Snapshots never expired, so storage grows without bound.
• Orphaned files never cleaned, from failed writes and abandoned compactions.
• Compaction conflicting with ingestion, causing repeated expensive retries.
• File count and file size unmonitored, so the problem is invisible until costs are investigated.
• Compaction disabled to save compute, which saves a little and costs far more in query spend.
• Very large tables compacted wholesale, when incremental partition-level compaction would be far cheaper.

Trade-offs

Compaction consumes compute continuously, and on a large estate that is a meaningful ongoing cost — paid against query savings that are real but distributed and less visible than the compaction job's own line item.

It also rewrites files, which interacts with time travel and with any consumer holding a snapshot reference, and aggressive snapshot expiry to control storage reduces the time-travel window that may be a compliance or recovery requirement.

The trade is a continuous maintenance cost and reduced retention flexibility in exchange for query performance that does not degrade. It is close to unavoidable for any table receiving streaming writes, and the honest framing is that it is not an optimisation but a running cost of the architecture — one that should be budgeted at adoption rather than discovered later.

Interview question

"Our lakehouse tables have got progressively slower over eighteen months with no schema or query changes. Tell me your first hypothesis, what you would measure to confirm it, and what you would put in place so that it does not recur — including what it will cost us."