Workload Class Isolation

Interactive queries and multi-hour batch pipelines have opposite requirements: one needs low, predictable latency at modest concurrency; the other needs maximum throughput and is indifferent to latency. Running them on shared compute means the second determines the experience of the first.

The decisive property of the lakehouse and modern warehouse model is that storage is shared and compute is not. Both read the same tables in the same format, on separate compute with separate sizing, scaling and failure domains.

Why it matters

An analyst facing unpredictable wait times stops using the platform, and the value lost there generally exceeds the cost of the idle capacity that isolation requires. Interference is also invisible in aggregate metrics: average query time looks acceptable while the p95 that analysts actually experience is dominated by whatever batch job is running.

Implementation patterns

• At least three classes, not two. Interactive — small, always warm, autoscaling on concurrency, latency-tuned, with a cache that unrelated work cannot evict. Batch — large, ephemeral, throughput-tuned, cheaper instance classes. And scheduled near-real-time pipelines, which are neither, and which degrade both if forced into either bucket. Most platforms discover the third class within a year.
• Independent autoscaling policies per class, since the signals differ: concurrency for interactive, queue depth for batch.
• Cost attribution per class and per team, which changes behaviour more than any technical control.
• Compaction and layout maintenance as a platform responsibility. Data layout remains shared even when compute is not, and small-file proliferation from a streaming writer slows every reader.
• Watch the catalogue. Metadata is shared and can bottleneck at extreme partition or object counts, which is the one resource isolation does not separate.

Industry example

Platforms such as Databricks and Snowflake are built around this separation, and the interesting failure is economic rather than technical. Elasticity makes it trivially easy to spend money: a warehouse that autoscales to satisfy a runaway dashboard refreshing every thirty seconds will do exactly that, silently.

The architectural response — per-workload budgets, query timeouts, result caching, auto-suspend — is typically added after the first surprising invoice rather than before it. Separation moves the constraint from capacity to cost, and cost is a softer constraint that is easier to ignore than a full disk.

Failure scenarios

• One cluster for everything, so batch determines interactive latency.
• A bigger cluster as the remedy, which raises the ceiling without removing the interference — a large job still saturates memory, spills, and evicts the cache interactive queries depend on.
• Night-only batch scheduling, which breaks when pipelines outgrow the night and forbids intraday refreshes.
• Query timeouts on the interactive workload, punishing the victim rather than the cause.
• No cost attribution, so the expensive workload has no owner.

Trade-offs

Isolation costs utilisation: each pool needs headroom and some of it sits idle. It also multiplies the configuration surface, since each class needs its own sizing, scaling and access policy.

The alternative — one pool, high utilisation — is genuinely cheaper in infrastructure and more expensive in everything else. The correct framing is that you are buying predictability, and predictability is what makes an analytical platform usable.

Interview question

"Analysts complain that the same dashboard takes four seconds some mornings and ninety seconds others. Your average query latency has not changed. What is happening, and what do you change first?"