A data platform runs interactive analyst queries alongside multi-hour batch pipelines on shared compute. Interactive latency becomes unpredictable. What is the architectural fix?

Why the alternatives fail

A bigger cluster raises the ceiling and does not change the interference: one large batch job still saturates memory, spills to disk, and evicts the cached data the interactive queries depend on. Night-only scheduling stops working the moment pipelines take longer than the night, and it forbids the intraday refreshes that analysts increasingly need. Query timeouts on the interactive workload punish the victim rather than the cause.

What the separation buys

The decisive architectural property of the lakehouse model is that storage is shared and compute is not. Both workloads read the same tables in the same open format, but they run on separate compute with independent sizing, scaling and failure domains.

• Interactive compute: small, always warm, autoscaling on concurrency, tuned for latency, with caching that is not evicted by unrelated work.
• Batch compute: large, ephemeral, tuned for throughput, cheaper instance classes, and free to consume all of its own resources without affecting anyone.
• A third class is usually needed within a year: scheduled near-real-time pipelines that are neither interactive nor batch, and forcing them into either bucket degrades both.

The parts people forget

• Data layout is a shared resource even when compute is not. Small-file proliferation from a streaming writer slows every reader, so compaction is a platform responsibility rather than a per-pipeline one.
• Metadata and the catalogue are shared and can bottleneck, particularly with very high partition counts.
• Cost attribution follows the isolation. Separate compute makes it possible to say which team's workload costs what, which changes behaviour more than any technical control.

The trade-off

Isolation costs utilisation: each pool needs headroom and some sits idle. That is the price of predictable interactive latency, and it is nearly always worth it — an analyst waiting an unpredictable amount of time stops using the platform, and the value lost there exceeds the cost of the idle capacity.