A platform's analytical queries scan far more data than they need. Which storage layout decisions fix this?

The decisions

1. Partition on the dimension queries filter by most. Almost always time, for event data. Partition pruning is the single largest reduction available — a query for one day should read one day.

2. Do not over-partition. Partitioning by hour and by several dimensions produces very many small partitions, and small files are the dominant cause of poor performance in file-based analytical storage. The metadata overhead and per-file cost exceed the pruning benefit.

3. Cluster or sort within partitions on the next most common filter. This enables file-level and block-level skipping using column statistics, delivering much of partitioning's benefit without its cardinality cost.

4. Columnar formats with useful statistics. Min/max per column per block lets the engine skip blocks without reading them. This is why format choice affects scan volume as much as layout does.

5. Compaction as a scheduled operation. Streaming ingestion produces many small files continuously. Without compaction, query performance degrades steadily — and this is the most common way a well-designed layout becomes slow.

The diagnostic

Measure bytes scanned per query relative to bytes needed. A large ratio points at a layout problem, and the breakdown says which:

• No pruning → the filter dimension is not the partition key.
• Pruning works, still scanning too much → clustering or file-level statistics are missing.
• Many small files → compaction is not running or partitioning is too fine.
• Wide scans of narrow queries → row-oriented storage, or columns not being projected.

The cardinality warning

Partitioning on a high-cardinality column is the classic mistake. Partitioning by user identifier produces millions of partitions, and the metadata cost alone makes queries slower than a full scan.

High-cardinality filters belong in clustering, not partitioning — which gives skipping without the partition explosion.

The property that ties layout to cost

In consumption-priced analytical systems, bytes scanned is the bill. Layout is therefore a cost control as much as a performance one, and the same change improves both — which is unusual and worth stating when justifying the work.