Columnar Formats and Why They Win
What changes when values of the same column sit next to each other on disk, why that makes compression an order of magnitude better, and the workloads where row storage is still correct.
A query reads three columns from a table with two hundred. In a row-oriented file it must read every byte of every row to find them, because the three values it wants are scattered across the record. In a columnar file it reads three contiguous regions and ignores the rest. That single difference, plus what it does to compression, is why analytical storage is columnar everywhere.
The layout
Parquet organises a file into row groups, typically 128 MB to 1 GB of rows. Within a row group, each column is stored as a contiguous chunk, subdivided into pages of around 1 MB. Metadata in the file footer records, for every column chunk, its byte offset, its encoding, and statistics: minimum, maximum, null count, and often a distinct count.
That structure supports two independent forms of skipping. Column projection reads only the chunks for requested columns. Predicate pushdown compares a filter against the min and max statistics and skips entire row groups or pages that cannot contain a match. A query filtering on a date range over a file sorted by date may touch a few percent of the bytes.
Why compression improves so much
Adjacent values in a column share a type and usually a distribution, which is exactly what compression exploits and what a row layout destroys by interleaving a string, a timestamp and a float.
The encodings do most of the work before any general-purpose compressor runs. Dictionary encoding replaces repeated values with small integer codes and is transformative on low-cardinality string columns such as country or status. Run-length encoding collapses repeats in sorted or clustered columns. Bit packing stores integers in the minimum number of bits their range needs. Delta encoding stores differences, which is very effective on sorted timestamps and monotonic identifiers.
Only then does Snappy, ZSTD or LZ4 compress the already-encoded bytes. Compression ratios of five to ten times against raw CSV are ordinary, and twenty times is common on wide tables with many low-cardinality columns.
Dictionary encoding is also directly useful at query time: a filter on a dictionary-encoded column can compare integer codes rather than strings, and can determine from the dictionary alone that a value is absent from the chunk.
When it breaks
Small files destroy the benefit and then some. Row group statistics and dictionaries are per file, so thousands of small files mean the metadata dominates and every query pays per-file open costs. On object storage, where each open is a network request with latency in the tens of milliseconds, this is usually the single largest performance problem in a data lake. Target file sizes in the hundreds of megabytes and compact regularly.
Point lookups are the wrong workload. Fetching one complete row means reading one value from every column chunk, which columnar layout makes worse rather than better. Serving a single record by key belongs in a row store or a key-value store, and building a serving path on Parquet is a recurring mistake.
Sort order determines whether statistics are useful. Min and max are only selective if the data is clustered on the filter column. In randomly ordered data every row group spans nearly the full range, every min-max test passes, and no skipping happens. The statistics still exist and buy nothing, which is why sorting on write matters more than most tuning.
Wide schemas cost even when unread. Every column chunk carries footer metadata, so a table with 2,000 columns has a large footer that must be parsed on every read regardless of projection. Very wide tables should be split by access pattern.
Nested data has a real cost. Parquet encodes nesting with definition and repetition levels, which is elegant and expensive: deeply nested structures inflate storage and slow decoding. Flattening frequently accessed fields into top-level columns is a standard and worthwhile denormalisation.
14 flashcards for this concept
Click a card to reveal the answer.