A data platform's interactive queries become unpredictable whenever a large pipeline runs. What is the fix, and what remains shared?

The fix

Separate compute per workload class over shared storage. Both read the same tables in the same format, on separate compute with independent sizing, scaling and failure domains.

Why alternatives fail: a bigger cluster raises the ceiling without removing interference — a large job still saturates memory, spills, and evicts the cache interactive queries depend on. Night-only scheduling breaks when pipelines outgrow the night and forbids intraday refreshes. Query timeouts on the interactive workload punish the victim.

The classes needed

• Interactive: small, always warm, autoscaling on concurrency, latency-tuned, with a cache that unrelated work cannot evict.
• Batch: large, ephemeral, throughput-tuned, cheaper instance classes.
• Scheduled near-real-time, which is neither and degrades both if forced into either bucket — most platforms discover this third class within a year.

What remains shared and must therefore be a platform responsibility

• Data layout. Small-file proliferation from a streaming writer slows every reader, and no individual writer is responsible for it — so compaction is a platform obligation rather than a per-pipeline one.
• The catalogue and metadata layer, which bottlenecks at very high partition or object counts and is the one resource isolation does not separate.
• Access control, which becomes more important with shared storage rather than less, since the perimeter is now a policy rather than a deployment.

The cost dimension that follows

Separate compute makes attribution automatic, which changes behaviour more than any technical control — teams that can see their own spend reduce it.

And elasticity makes it trivially easy to spend money: a warehouse autoscaling to satisfy a runaway dashboard will do exactly that, silently. Per-workload budgets, statement timeouts, result caching and aggressive auto-suspend are the controls, and they are typically added after the first surprising invoice rather than before.

The trade-off to state

Isolation costs utilisation, since each pool needs headroom and some sits idle. It is nearly always worth it, because an analyst facing unpredictable wait times stops using the platform — and that loss exceeds the cost of the idle capacity.