A design platform's synchronous editing experience and its asynchronous export rendering share a compute cluster. Exports occasionally saturate it and the editor becomes unusable. What would you change?
Show the full answer Hide the answer
The diagnosis
Two workloads with completely different characteristics and completely different criticality are competing for the same resource with no arbitration:
- Editing is latency-sensitive, interactive, and the product's core value. A 200 ms delay is noticeable.
- Export rendering is throughput-oriented, bursty, individually expensive, and tolerant of seconds or minutes of delay.
Sharing a cluster means a burst of exports — a customer bulk-exporting a large document set — consumes CPU and memory that the editor needs, and the most important workload is degraded by the least important one.
What to change
1. Separate clusters, not just separate queues. Queues arbitrate ordering; they do not prevent one workload from consuming the memory and CPU the other needs. For workloads this different, physical separation is the reliable answer and the additional cost is small relative to the outcome.
2. Resource requests and limits on every workload, enforced by admission policy. A workload that does not declare its needs is one the scheduler cannot reason about, and it is the root of most noisy-neighbour incidents.
3. Priority classes with preemption. If separation is not possible immediately, ensure the interactive workload can evict batch work under pressure. This is the cheapest interim mitigation.
4. Bound the export workload explicitly. Per-customer concurrency limits on rendering, so a bulk export is spread over time rather than admitted all at once. A queue with a bounded worker pool converts a spike into a slower completion, which is exactly the right trade for asynchronous work.
5. Classify by duration at the entry point. A render predicted to exceed the synchronous budget becomes a job; anything else stays inline. This prevents the gradual re-acquisition of long work by the interactive path, which is how the shared-cluster situation typically arose.
What not to change
Do not simply add capacity. It restores service and guarantees recurrence at the next scale, because absent isolation, capacity is just a bigger shared pool for the burst to consume.
Do not make exports synchronous to "control" them. That moves the problem into the user's request path and makes it worse.
The generalisable principle
Isolate by failure policy and criticality, not by team or by technology. Two workloads belong on shared infrastructure when they can tolerate each other's worst behaviour. When one is interactive and critical and the other is batch and expensive, they cannot — and the sharing is a latent incident waiting for a customer large enough to trigger it.