One customer's batch job saturates a shared service and degrades everyone. Rate limiting them fixes it, until the next customer does the same. What is the structural answer?

Why per-customer rate limits keep failing

A static limit is set from what that customer was doing, not from what the service can serve. It is reactive — you discover each new offender in production — and it is wrong in both directions: too low during that customer's legitimate peak, too high when several customers are busy simultaneously.

The underlying problem is that a rate limit is an absolute control on a shared, variable resource.

The structural answer, in layers

1. Per-tenant concurrency limits, not just rate limits. Concurrency is the resource that actually runs out. A cap on in-flight requests per tenant means one tenant can occupy at most its share of the pool, regardless of how fast it sends. This is the single highest-value change.

2. Adaptive admission control. Instead of a configured limit, infer capacity from measured latency and shed when the service is approaching saturation — controllers modelled on TCP Vegas do this. It adapts to the service's real capacity, which changes with deployments, data volume and downstream health.

3. Fair queueing between tenants. Under contention, serve tenants round-robin or weighted rather than first-come-first-served, so a tenant sending ten times the volume gets its share rather than ten shares.

4. Shuffle sharding. Assign each tenant a random subset of workers. A pathological tenant degrades only the small fraction of tenants sharing its full subset — and with retries against their other assigned workers, most of those see no impact.

5. Workload separation. Batch and interactive traffic on separate capacity, with batch explicitly lower priority and shed first. Very often the real fix, because the two have completely different latency requirements and should never have shared a pool.

6. Dedicated capacity for the largest tenants — where multi-tenancy economics stop paying, and an honest answer.

What I would do first

Layers 1 and 5: per-tenant concurrency caps, and separating batch from interactive. Together they address most of the observed problem within a sprint, and neither requires re-architecture.

What a strong answer adds

Per-tenant resource attribution as a prerequisite for all of it. Without knowing which tenant is consuming what, every incident begins with elimination. With it, the noisy neighbour is identified in minutes and the limits can be set from evidence rather than from the last outage.