Pods have stopped scheduling on a cluster that has plenty of free CPU and memory. What do you check?

What is being tested

Whether you recognise IP address exhaustion, which presents as a capacity problem and is an address planning problem.

The most likely cause

Subnet address exhaustion. In most cloud container networking modes, every pod consumes an IP address from the VPC subnet. A subnet sized during the VM era — when a node consumed one address and you had thirty nodes — is exhausted by a cluster where each node runs thirty pods.

The failure is opaque: pods sit pending, and the message is about node capacity or IP allocation in terms an application team does not recognise as an address problem. Free CPU and memory make it look like a scheduler bug.

The full checklist, in order

1. Free IPs in the subnets the cluster uses. Check each subnet, per availability zone — one may be exhausted while others have room, which produces the confusing symptom of pods scheduling sometimes.
2. Pods-per-node limits. Many networking plugins cap pods per node based on the number of network interfaces and addresses the instance type supports. A large instance may still be limited to a modest pod count.
3. Resource requests against allocatable, not against total. Nodes reserve capacity for the kubelet and system daemons, so "free memory" as reported by the host overstates what is schedulable.
4. Taints, affinities and topology spread constraints preventing placement despite available capacity.
5. Quotas and limit ranges in the namespace.
6. Persistent volume zone affinity — a pod bound to a volume in one zone cannot schedule in another.

The fixes for address exhaustion

• Add a secondary CIDR range to the VPC and additional subnets for pods. Usually the fastest remedy.
• Use an overlay network so pods have their own address space and do not consume VPC addresses. Costs some direct reachability and observability, and is the standard answer at high density.
• Increase prefix delegation or address density where the platform supports assigning blocks rather than individual addresses per node.
• Larger subnets, which is free if done at design time and disruptive afterwards.

The design lesson

Size subnets for the density you will have, not the density you have. Address space costs nothing and running out is painful. Leave contiguous gaps between allocations so a range can be grown adjacently rather than fragmenting the space, and document the allocation scheme somewhere findable — that document is what prevents the overlap that blocks a future peering.

The general habit

When a symptom points at capacity and the obvious capacity is available, look for the resource nobody is monitoring. Addresses, file descriptors, connection pool slots, inodes, ephemeral ports, and per-account service limits all produce the same shape: something is exhausted, and it is not the thing on the dashboard.