advanced 2 min answer

An analytics platform's query costs are growing faster than its data. What are the drivers and controls?

analytics-costbytes-scannedmaterialisationgovernancetwitchdebugging
Show the full answer Hide the answer

The drivers

1. Queries scanning far more than they need. Missing partition filters, SELECT * over wide tables, and layouts that do not match access patterns. In consumption-priced systems, bytes scanned is the bill.

2. Dashboards refreshing on a schedule regardless of use. A dashboard nobody opens, refreshing every fifteen minutes, is a continuous cost with no consumer.

3. Repeated computation of the same aggregate by many queries, where one materialised result would serve all of them.

4. Everything materialised as tables, rebuilding the world nightly — or everything as views, pushing enormous cost to query time. Both extremes are expensive in opposite ways.

5. Small file accumulation, making every query pay file-handling overhead.

6. Exploratory queries against raw data where a curated model would answer the same question for a fraction of the scan.

The controls

  • Attribution first. Cost per query, per user, per dashboard, per team. Without it, optimisation targets the wrong thing, and the intuitive target — the most expensive single query — is frequently a small share of spend while a cheap query run ten thousand times dominates it. Rank by cost × frequency.
  • Query cost limits and warnings, so an expensive query is refused or flagged rather than discovered on the bill.
  • Partition filters required on large tables, enforced rather than encouraged.
  • Materialise what is repeatedly computed, and only that — decided from usage data.
  • Expire unused dashboards and scheduled queries automatically. A refresh with no viewer in ninety days is waste.
  • Compaction and layout maintenance as scheduled work.
  • Cost visible to the team that incurs it, daily, in their own space.

The governance framing

Unit cost, not absolute cost. Cost per query, per active analyst, per gigabyte of source data. Absolute analytics spend should grow with the business; unit cost should not — and that reframing is what makes the conversation productive rather than adversarial.