On a shared cluster serving twelve teams, one team's new operator creates a custom resource per user session, reaching about 50,000 objects in a day. Other teams start seeing slow deploys and intermittent API timeouts. What is happening inside the cluster?

Second by second, what happens

Every Kubernetes object lives in etcd, and etcd is a single consistent store shared by the whole cluster. 50,000 objects is not, by itself, fatal. The damage comes from three multiplications.

• Watch fan-out. Every controller watching that resource type receives every change. The API server maintains these watches and serialises objects to each watcher, so cost scales with objects × watchers, not with objects.
• List calls. A controller that lists all objects of a type on resync pulls 50,000 objects into API server memory and then over the network, repeatedly. A poorly written operator that lists rather than watches turns this into a continuous load.
• etcd write amplification. Frequent updates to many objects grow the etcd database and its revision history. Compaction and defragmentation become necessary, and defragmentation makes etcd unavailable for the duration, which is a cluster-wide stall.

The symptom the other teams see is the API server's latency, which is the front door for every deploy, every controller and every kubectl command in the cluster.

Where it amplifies

etcd has practical size limits measured in gigabytes, with the default quota at 2 GB in many distributions and the recommended ceiling around 8 GB. When the quota is exceeded, etcd goes read-only and the entire cluster stops accepting changes. That is the cliff: behaviour degrades gradually and then stops completely.

What the user sees

Nothing, at first. Running workloads are unaffected because the data plane does not consult etcd per request. The visible damage is to change: deploys hang, autoscaling stops reacting, certificate renewals queue, and an incident response that needs a deploy discovers it cannot have one.

What stops it

• Resource quotas per namespace, including object counts, not only CPU and memory. Object count quotas are the specific control for this and are almost never set.
• A policy that custom resources represent infrastructure, not application state. Session data belongs in a database. The etcd API is a configuration store with a watch mechanism, not a key-value database, and treating it as one is the root design error here.
• Monitoring etcd database size and object counts by type, with an alert well below the quota. This is a boring dashboard that prevents a cluster-wide outage.
• Separate clusters for workloads with unusual control-plane behaviour, which is the expensive answer and is correct when a team's legitimate design needs object counts that threaten shared capacity.

What would have to be true for it to self-heal

Only that the objects have a TTL or an owner reference that garbage-collects them. Without one, they persist until somebody deletes them, and mass deletion is itself a write storm that must be rate-limited. The recovery is slower than the damage, which is the usual asymmetry in shared control planes.

When not to move the team off the shared cluster

The reflex after an incident like this is a cluster per team, which multiplies upgrade work by twelve and gives each team a control plane nobody is expert in. Object quotas and a design review fix this case for a fraction of that cost. Split the cluster only when a team's legitimate workload genuinely needs control-plane behaviour the shared cluster cannot safely offer, such as custom admission webhooks in the critical path or a different Kubernetes version.