A service's p99 is 300 ms against a p50 of 12 ms. Average CPU across its pods sits at 25% of the configured limit, there are no garbage-collection pauses, every downstream call returns in under 20 ms, and the nodes are not saturated. Where is the time going?

The first three signals

1. Throttling counters per container, not node CPU: container_cpu_cfs_throttled_periods_total against container_cpu_cfs_periods_total. Any sustained ratio above a few percent on a latency-sensitive service is the answer.
2. The shape of the distribution. A p99 25 times the p50 with a flat average is a stop-start pattern, not a slow-work pattern.
3. Runnable thread count against the CPU limit, because the gap between them is the mechanism.

The diagnosis

A Kubernetes CPU limit is not a speed setting. It is translated into a Linux CFS bandwidth quota: a number of microseconds of CPU the cgroup may consume per enforcement period, which is 100 ms by default. A limit of 500 m means 50 ms of CPU time per 100 ms window.

Now add a runtime with several runnable threads - a JVM with a thread pool, a Go process with GOMAXPROCS set to the node's core count, any worker pool sized from nproc. Eight threads running in parallel consume the 50 ms budget in roughly 6 ms of wall-clock time. The cgroup is then frozen for the remaining 94 ms of the period. A request that arrives in that frozen window waits for the next period before it can make progress, which is exactly the tens-to-hundreds of milliseconds the p99 is showing.

Average utilisation reports 25% because the average is taken over whole periods - the process is at 100% for 6 ms and 0% for 94 ms, and the mean hides the stall completely. This is why the usual dashboards exonerate the service.

The misleading signal

Node CPU. The node has spare capacity throughout, which makes "we are not CPU-bound" feel like a measured fact. The constraint is not the node's capacity but the cgroup's permission to use it, and nothing at node level expresses that.

The fix, in order

1. Match parallelism to the quota. If the limit is 500 m, a runtime that believes it has 64 cores will always burn its budget in a burst. Set GOMAXPROCS, the JVM's active-processor count, or the thread-pool size from the container's limit rather than from the node - several runtimes now do this automatically when the limit is visible, and older ones do not.
2. Raise the limit, or remove it. Keeping the request preserves scheduling and fair-share behaviour; removing the limit removes the quota that creates the stall. This is the standard recommendation for latency-sensitive workloads and it is not free - see the trade-off below.
3. Shorten the period where the platform allows it, so the same quota is spread over more, smaller windows and the worst-case stall shrinks proportionally.
4. Check the kernel. The CFS quota accounting defect that throttled cgroups well below their allowance was fixed in the 4.18-to-5.4 era; a cluster on an older kernel throttles more than the arithmetic predicts.

What it costs

Removing CPU limits means a pod can consume whatever the node has spare, so one misbehaving service can degrade its neighbours. The controls that make that acceptable are honest CPU requests (so the scheduler does not oversubscribe), priority classes so the important workloads win contention, and node pools that separate latency-sensitive services from batch. Memory limits stay - memory is not compressible and an unbounded pod takes the node down.

When this is the wrong answer

If throttled periods are at or near zero, this is not the cause and the investigation should go back to lock contention, connection pool waits or a dependency's tail. The diagnosis here rests on one counter, and it is cheap to check before changing anything.