A team keeps three years of event data as gzipped CSV in object storage because it is simple and anything can read it. The table is now about 500 GB compressed and the daily dashboard query reads four of the 60 columns. What has the simplicity actually cost them?

What the format is costing

Two properties, and both are invisible until the data gets big.

Gzip is not splittable. A compressed stream must be decoded from its start, so one file can be read by exactly one worker. A 20 GB gzipped file is a single-threaded job no matter how much compute is attached. Adding nodes does nothing, which is the confusing part: the cluster is idle and the query is slow.

CSV is row-oriented text with no column boundaries known in advance. Reading four of 60 columns still means decompressing every byte and parsing every field to find the commas. In a columnar format those four columns are stored contiguously and the other 56 are never read. For a 60-column table where the four are of ordinary width, that is roughly an order of magnitude fewer bytes off storage, before compression differences are counted.

There is a third, quieter cost: CSV has no types and no statistics, so the engine infers a schema on every read, cannot prune by minimum and maximum, and will happily change its mind about whether a column is an integer when a new file arrives with a blank in it.

Why the other options fail

• "Object storage charges more for text." Storage is priced by bytes and requests; the format is not a billing dimension. The bill arrives at the query engine, which is why the storage line item looks fine while the compute line item does not.
• "Nested fields were lost." CSV does flatten nested data, and that is a real constraint, but it is a modelling limitation rather than the cause of the cost described here.
• "The files are too large." File size matters, and the opposite problem — millions of tiny files — is the more common failure. Neither is the mechanism at work: the format is.

What it costs to fix, and when not to

Converting to a columnar format with a splittable codec is a one-off backfill plus a change to the writer. What you give up is real: a schema that now has to be managed and evolved, files that a human cannot read with less, and a dependency on a library for every consumer that previously needed none.

Stay on CSV while the dataset fits comfortably on one machine and is read whole — a few gigabytes, a nightly load into a database, an interchange file for a partner who has asked for CSV. It flips when queries become selective or when the table outgrows one reader, which in practice is somewhere in the tens of gigabytes. Converting at 500 GB works; the reason to do it earlier is that the conversion gets no cheaper and the queries get no faster while you wait.