A lakehouse team raises its compaction target file size from 128 MB to 1 GB and switches the write codec from Snappy to Zstandard. Query cost falls. What did they give up, and when does that bill arrive?

What is gained, quantified

• Fewer objects to open. Planning and reading cost has a fixed per-file component: a metadata read, a GET, a connection. Going from 128 MB to 1 GB files cuts the object count for the same data by roughly 8x, and on request-priced object storage that is a direct line-item reduction.
• Better compression. Zstandard typically produces files 10–30% smaller than Snappy on the same data at moderate levels, which reduces both storage and the bytes a scan-priced engine bills for.

Datadog's writeups of Husky, its event store, describe exactly this economy: compaction of many small ingest fragments into larger ones to reduce object-store fetches, with bounded row groups so the merge's memory stays predictable.

What is paid

• Compaction CPU and write amplification. Every byte is now written at least twice, and Zstandard costs more CPU per byte to compress than Snappy. The compaction cluster grows.
• Decompression on every read. Zstandard decompresses fast, but not free; a scan-heavy, CPU-bound workload can lose more in decode than it saves in bytes.
• Coarser pruning. Statistics are per row group and per file. A 1 GB file is a coarser unit to skip than a 128 MB one, so a selective query that used to skip 90% of files may now read several times more data unless the data is sorted or clustered on the predicate.
• Latency to visibility for streaming writes, if compaction is in the path before data is queryable.

When the bill arrives

Not on the dashboard queries, which are large scans and get better. It arrives on the selective lookups: the support engineer querying one customer id, the reprocessing job reading one hour. Those are the queries whose cost is set by pruning, and pruning just got coarser. The symptom is a widening gap between median and p95 query cost with no change in data volume.

How to keep the option to reverse

Target file size is a property of the compaction job, not of the data, so it can be changed going forward at any time — but already-compacted files have to be rewritten to go back, and that rewrite costs the same as the original compaction. Before changing both variables at once, change one, measure selective-query bytes scanned as well as total, and keep the sort or cluster key aligned to the most common predicate so larger files stay skippable.

When not to do this at all

If the table is append-only, read whole, and already produces files in the hundreds of megabytes, compaction tuning is optimising something that is not the constraint. The small-file problem is worth attacking when the median file is measured in kilobytes, which is the signature of a streaming writer committing every minute — not when it is already in the right order of magnitude.