CFS Quota Throttling

A service reports a p50 of 12 ms and a p99 of 300 ms. Average CPU across its pods is a quarter of the configured limit, the nodes have spare capacity, garbage collection is quiet and every downstream call returns in under 20 ms. Nothing in the usual dashboards can account for 288 ms.

A Kubernetes CPU limit is not a speed setting. It is translated into a Linux CFS bandwidth quota: a budget of CPU microseconds the cgroup may consume per enforcement period, 100 ms by default. A limit of 500 m means 50 ms of CPU per 100 ms window. A runtime with eight runnable threads consumes that budget in roughly 6 ms of wall-clock time, and the cgroup is then frozen for the remaining 94 ms. Any request that arrives in the frozen portion waits for the next period before it can make progress.

The average reports 25% because it is taken over whole periods. The process is at 100% for 6 ms and 0% for 94 ms, and the mean hides the stall entirely - which is why this is diagnosed late and often misattributed to a dependency.

Why it matters

It is one of the few performance problems where the standard metric actively exonerates the cause. Teams spend days on downstream latency, connection pools and garbage collection because CPU "is not the problem", and the counter that would have answered it in a minute is not on any default dashboard.

It also scales with what platforms encourage. Limits are set as hygiene, often by an admission policy, and the smaller the pods the smaller each quota and the larger the effect of one burst. A sensible-sounding governance rule produces a latency floor nobody can find.

Implementation patterns

• Match runtime parallelism to the quota. Set GOMAXPROCS, the JVM's active processor count, or a worker-pool size from the container's limit rather than from the node's core count. Several runtimes now do this automatically when the limit is visible; older versions and hand-rolled pools do not.
• Remove CPU limits for latency-sensitive services and keep requests. Requests preserve scheduling and proportional share under contention; the limit is what creates the quota. This is now common guidance and it is not free - see the trade-offs.
• Keep memory limits. Memory is not compressible, so an unbounded pod takes the node with it. The asymmetry is the whole point: limit the resource that cannot be reclaimed, be careful with the one that can.
• Shorten the enforcement period where the platform exposes it, so the same budget is spread over more windows and the worst-case stall shrinks in proportion.
• Alert on the ratio of throttled periods to total periods per container, not on node CPU - a few percent sustained matters for a user-facing service.

Industry example

The mechanism is documented in the Linux kernel's CFS bandwidth control documentation and in Kubernetes' own resource-management pages, which describe limits as a quota per period rather than as a rate. The empirical history is equally instructive: a kernel accounting defect throttled cgroups that had not exhausted their allowance, was fixed in kernels released between 2018 and 2019 (4.18 through 5.4), and generated a long tail of public reports in which teams measured throttling far above what the arithmetic predicted.

The pattern worth carrying is procedural: in nearly every case the resolution was to remove CPU limits from the affected service while keeping requests, and in nearly every case the change was first resisted as unsafe - which is the conversation to prepare for.

Failure scenarios

• Tail latency at low utilisation, the canonical presentation, routinely misdiagnosed as a slow dependency.
• A runtime that sizes its thread pool from the node, so a 64-core node gives a 500 m container 64 threads racing through its budget.
• Autoscaling that will not trigger, because the horizontal autoscaler reads utilisation against the request and a throttled pod reports low utilisation - the scaler stays quiet exactly when latency is worst.
• Startup timeouts. Initialisation is CPU-intensive, so a tight quota can push a container past its readiness deadline and into a kill loop - an outage at deploy time rather than a slowdown.
• Over-provisioning as compensation: replicas raised until p99 looks acceptable, paying several times the compute for a configuration change.

Trade-offs

The honest position: limits are a blunt instrument for noisy-neighbour protection, and requests plus priority classes plus pool separation do the same job without the stall. Keep limits where utilisation matters more than latency - batch, cron, development clusters.

When not to use it

Removing CPU limits is not a universal recommendation. On a shared cluster with untrusted or unpredictable workloads, limits are the mechanism that stops one tenant consuming a node, and removing them without honest requests, priority classes and node sizing just moves the problem to a neighbour.

And do not reach for this diagnosis without the counter. If throttled periods are near zero, the stall is somewhere else, and the time spent arguing about limits is time not spent on the real cause. The counter is the entire evidential basis for the change; collect it first.

Interview question

Q: A payments service shows a p99 of 300 ms with average CPU at 25% of its limit and no slow dependencies. Walk me through your diagnosis, the change you would make, and how you would answer a platform engineer who says removing CPU limits is unsafe.

What a strong answer covers: the quota-per-period mechanism and the arithmetic from limit to stall; why the average hides it and node CPU exonerates it; the throttled-periods counter as the evidence; matching runtime parallelism to the quota as the least invasive fix; removing the limit while keeping the request, with priority classes and honest requests as the compensating controls; keeping memory limits because memory is not compressible; and the autoscaling interaction, which is often the more expensive symptom.

Quick check

Quiz: Why does a container with 25% average CPU utilisation stall for tens of milliseconds at a time? Because the limit is a quota per 100 ms period, so a multi-threaded process can exhaust the whole budget in a few milliseconds and is frozen for the rest of the period - and the average over whole periods hides it.

Flashcard: Which two settings should move in opposite directions on a latency-sensitive container? Keep the memory limit, because memory cannot be reclaimed; relax or remove the CPU limit, because the quota is what produces the stall.