Object Count Quota

Most multi-tenancy controls are about compute: CPU requests, memory limits, node pools. They do not protect the thing that actually fails first in a shared cluster, which is the control plane. A team that stays inside its CPU quota can still create hundreds of thousands of objects, and every one of them lives in the same etcd cluster as everybody else's deployments.

An object count quota is the missing limit: a cap on how many of each resource kind a namespace may hold, enforced at admission, which turns a cluster-wide outage into one team's error message.

Why it matters

The cost of an object is not its size. It is objects multiplied by watchers: every controller watching a resource type receives every change, and the API server serialises the object to each one. A controller that lists instead of watching pulls the whole set into API server memory on every resync. So 50,000 objects of a type watched by six controllers is a very different load from 50,000 objects nobody watches.

Underneath is a hard cliff. etcd's database quota defaults to 2 GB in many distributions, with a recommended ceiling around 8 GB, and exceeding it puts etcd into a read-only alarm state that stops the cluster accepting any change. Defragmentation, which large databases eventually need, also stalls the cluster while it runs.

Implementation patterns

• Set count/<resource>.<group> quotas per namespace for custom resources as well as core ones. Most teams set CPU and memory quotas and stop.
• Default them at namespace creation in the platform's provisioning path, so a new tenant arrives with limits rather than acquiring them after an incident.
• Alert on etcd database size and on object counts by kind, with thresholds set as a fraction of the quota rather than as absolute numbers.
• Require owner references or a TTL on anything created programmatically, so objects are garbage-collected when their parent goes rather than accumulating.
• Review new custom resource definitions for cardinality: one object per cluster, per service and per user session are three different orders of magnitude, and the third is usually a design error.

Industry example

The pattern is standard Kubernetes multi-tenancy guidance, documented since resource quotas for object counts were generalised in the 1.9 release (2017), and the failure it prevents is routine enough that managed Kubernetes providers document etcd size limits prominently in production guidance and several publish per-cluster object count ceilings. Operators that model per-request or per-session state as custom resources are a recurring cause of control-plane incidents reported in Kubernetes community postmortems, precisely because the design looks idiomatic while the cardinality is wrong.

Failure scenarios

• Gradual API latency growth across every team, attributed to "the cluster being slow", with the cause invisible unless someone counts objects by kind.
• etcd read-only alarm, which stops all deploys, autoscaling reactions and certificate renewals cluster-wide while running pods continue serving.
• Mass deletion as a second incident: removing 50,000 objects is itself a write storm that must be rate-limited.
• Quota set too low, blocking a legitimate workload at 3am with an error message that reads as a platform fault rather than a policy decision.

Trade-offs

Quotas convert a shared, silent, cluster-wide risk into a visible, per-tenant error. The cost is that someone must own the numbers: tenants hit limits, ask for increases, and the platform team must answer quickly or become the bottleneck they were trying to avoid. Set too low, the quota is a support queue; set too high, it does not bind before etcd does. A workable default is one order of magnitude above a tenant's current peak, reviewed when they approach it.

When not to use it

On a single-team cluster, this is paperwork. The team that would breach the quota is the same team that would be paged, and the feedback loop is already tight. It is also the wrong control when a tenant's legitimate design genuinely needs high object counts: then the answer is a separate cluster for that workload, accepting the upgrade and operational cost, rather than a quota that blocks them from doing their job. Quotas protect against accidents and bad designs; they are not a substitute for isolation when the requirement is real.

Interview question

Q: A tenant asks for their custom resource quota to be raised from 10,000 to 500,000 because their operator models each active user session as an object. How do you answer?

What a strong answer covers: refusing the number and addressing the design, because etcd is a configuration store with a watch mechanism rather than a session database · quantifying the effect through watchers and etcd size, not just count · offering the alternative, which is application state in a database with a small number of custom resources describing the deployment · what you would do if they insist and are a critical team, namely a separate cluster with its own control plane · the platform principle that the answer to "raise the limit" is sometimes "the limit found a bug".

Quick check

Quiz: Why is CPU and memory quota insufficient for multi-tenant isolation? Because it does not bound control-plane load, and etcd exhaustion stops the whole cluster while every tenant is inside its compute limits.

Flashcard: What is the real multiplier on object cost? — Objects × watchers: every controller watching the type receives every change.