A data platform's ad-hoc queries, scheduled pipelines and machine-learning training compete for the same capacity. How should they be isolated?
Show the full answer Hide the answer
Why they must be isolated
They have completely different characteristics and completely different criticality:
- Scheduled pipelines must complete within a window and feed everything downstream.
- Ad-hoc analysis is interactive, unpredictable, and occasionally pathological — one analyst's unbounded query can consume enormous capacity.
- Machine-learning training and backfills are large, long-running, and tolerant of delay.
Sharing capacity first-come means a backfill starves the nightly run, or an exploratory query delays a production pipeline. First-in-first-out is not a scheduling policy; it is the absence of one.
The isolation mechanisms
1. Separate compute pools per workload class, which is the reliable answer where the platform supports it. Pipelines, interactive analysis and training on separate capacity.
2. Priority classes with preemption where pools are shared, so a scheduled pipeline can preempt a backfill.
3. Per-user and per-team concurrency and cost limits, so one analyst cannot occupy the cluster.
4. Query cost limits with fast rejection, so a pathological query is refused rather than admitted and allowed to consume.
5. Backfills explicitly resourced separately, since a reprocessing job replaying months of history will consume the same capacity as the regular schedule unless prevented — and this is the most common contention incident on a data platform.
6. Time-of-day scheduling for flexible work, moving it away from the peak.
The measurement that reveals the problem
Queue time by workload class. Aggregate cluster utilisation looks healthy while pipelines are waiting behind an analyst's query — the same failure as any aggregate metric hiding a concentrated problem.
Alert on outliers: any pipeline whose queue time exceeds its budget, any user exceeding a cost threshold.
The organisational addition
Publish the isolation policy. Analysts should know that their exploratory query runs on a pool with a cost limit, and that a large job needs to be submitted differently. An undocumented policy is experienced as arbitrary throttling, and it generates escalations rather than adaptation.