Small File Problem
The degradation that follows from many tiny files — metadata operations dominating, compression failing, and columnar formats losing their advantage.
Streaming ingestion and frequent micro-batches produce a file per commit. After a year of one-minute commits a table has half a million files, most of them a few kilobytes.
Three effects compound. Every query pays a listing and open cost per file, which at that count exceeds the time spent reading. Columnar formats organise data into row groups sized for efficient scanning, and a file smaller than one row group gets none of that benefit — compression ratios collapse and per-column skipping stops working. And object storage request charges become a visible line item.
Compaction — rewriting many small files into fewer larger ones, typically a few hundred megabytes each — is the remedy, and it is a scheduled maintenance job that someone has to own. Table formats make it safe by committing the rewrite atomically, so readers are unaffected.
The reason it bites so often is that nothing fails. The table works, queries return correct results, and the cost and latency rise gradually enough that no single week looks wrong.