The analytics bill has tripled in six months with no corresponding growth in users or data volume. Where do you look?
Show the full answer Hide the answer
Get cost per query, attributed
Almost every analytical engine can report bytes scanned and compute time per query, with the identity that ran it. Without that attribution the investigation is guesswork; with it, the answer usually appears within an hour.
Group by identity and by scheduled versus interactive. The distribution is characteristically extreme: a small number of queries account for most of the spend.
The four findings, in descending order of frequency
A dashboard refreshing too often. Set to five minutes, running a full table scan, built by someone who left, watched by nobody. This is the single most common cause and it is usually a material share of the bill.
A pipeline that stopped pruning. A predicate wrapped in a function, a partition column compared against a different type, or a filter moved onto a derived column — each silently defeats partition pruning while returning correct results. Cost rises fifty-fold and nothing errors.
Small files accumulating. Streaming ingestion or frequent micro-batches produce one file per commit. Metadata listing begins to dominate read time, compression ratios collapse, and object storage request charges appear. Nothing fails; it just gets slower and more expensive every week.
Concurrency scaling absorbing a queue. Additional clusters spin up automatically to remove contention, converting a latency problem into a cost one. Nobody complains any more, which is why it goes unnoticed.
The fix that outlasts the investigation
Per-team cost visibility, monthly, broken down by query and by resource. It changes behaviour faster than any optimisation programme, because the team that owns the five-minute dashboard is the only one who knows whether it is still needed.
Then structural separation: distinct compute for pipelines, dashboards and ad-hoc analysis. Sharing one pool means an analyst's accidental cross join delays the morning refresh, and it makes the attribution above much harder.