Bytes Scanned

In a consumption-priced analytical platform, a query's cost is a function of how much data it reads. That makes bytes scanned the unusual metric that is simultaneously the performance measure, the capacity measure and the invoice line.

It also means layout decisions and cost decisions are the same decisions, which is rare and worth stating explicitly when justifying investment in partitioning, clustering and compaction: the work improves both, and the improvement compounds across every query.

The diagnostic ratio

Bytes scanned relative to bytes actually needed. A large ratio localises the problem:

• No pruning at all → the filter dimension is not the partition key.
• Pruning works but scans are still large → clustering or file-level statistics are missing.
• Scanning entire rows for a few columns → row-oriented storage, or columns not being projected.
• Many small files → compaction not running, or partitioning too fine.

Implementation patterns

• Partition on the dimension most queries filter by, usually time for event data — the largest single reduction available.
• Cluster or sort within partitions on the next most common filter, which gives block-level skipping without partition explosion. High-cardinality filters belong here, not in partitioning.
• Columnar formats with per-block statistics, so the engine can skip without reading.
• Require partition filters on large tables, enforced rather than encouraged.
• Query cost limits with warnings, so an expensive query is refused or flagged rather than discovered on the invoice.
• Materialise what is repeatedly computed, decided from usage data rather than from intuition.

Industry example

Analytics platforms whose costs grow faster than their data almost always find the same drivers: queries with missing partition filters, SELECT * over wide tables, dashboards refreshing on a schedule regardless of whether anyone opens them, and the same aggregate recomputed by many queries.

The intervention that matters most is attribution — cost per query, per user, per dashboard, per team. Without it, optimisation targets the wrong thing, and the intuitive target is misleading: the single most expensive query is frequently a small share of spend while a cheap query run ten thousand times dominates it. Rank by cost multiplied by frequency.

The governance frame that makes the conversation productive is unit cost rather than absolute cost — cost per query, per active analyst, per gigabyte of source data. Absolute spend should grow with the business; unit cost should not.

Failure scenarios

• Optimising the largest single query rather than the largest total contribution.
• Partitioning on a high-cardinality column, where metadata cost alone makes queries slower than a full scan.
• No cost attribution, so nobody owns the number and every reduction reverses.
• Scheduled refreshes for dashboards nobody opens, which is continuous cost with no consumer.
• Layout maintenance treated as optional, so performance and cost degrade together with elapsed time.

Trade-offs

Aggressive layout optimisation costs engineering effort and compaction compute, and it optimises for the current access pattern — a pattern change can make a carefully-tuned layout wrong, and some formats make repartitioning expensive.

The mitigation is to partition on something stable, usually time, and to use clustering for the dimensions that vary — which is both more forgiving and cheaper to change.

Interview question

"Your analytics bill has doubled while data volume grew 20%. Walk me through how you would find the cause, and tell me which optimisation you would expect to deliver the most."