CIDR Exhaustion

A platform team plans for 400 nodes in a /16 virtual network. Each node is allocated a /24 for its pods. The cluster stops at 256 nodes, because a /16 contains 256 /24 prefixes, and it stops earlier than that in practice because load balancers, databases and other environments take subnets from the same space.

The failure is not an address shortage. It is a prefix shortage, and the two are different: those 256 nodes host at most 110 pods each, so the design reserves 256 addresses to use 110 and runs at under 43% utilisation by construction.

Why it matters

Exhaustion presents as pods stuck pending or nodes failing to join, during a scale-out, under load. Nothing is broken and nothing alerts, because no resource is at a threshold anyone configured; the allocator simply has no prefix to hand out. It is one of the few capacity failures with no gradual signal at all.

Recovery is slow in a way that matters. Adding a secondary range to a virtual network, creating subnets, updating route tables and replacing nodes is hours of work with a change-approval step, and it has to be done in the middle of the event.

Implementation patterns

• Size the per-node prefix to real density, not maximum density. Measured density is frequently 20 to 40 pods against a configured maximum of 110, and a /26 instead of a /24 is a 4x improvement for no new technology.
• Use prefix delegation where the CNI supports it: small prefixes, such as /28 blocks of 16 addresses in the AWS VPC CNI's implementation, assigned as pods are scheduled, so allocation tracks density. It needs contiguous free space, which a fragmented network may lack.
• Reserve space at design time with a documented plan. One range per region per environment, with the cluster's pod space carved from a block nothing else may touch, and the plan held outside the cluster's own tooling.
• Keep an overlay option open. Pod addresses drawn from a non-routable range remove the cluster from the estate's plan entirely, at the cost of direct reachability, straightforward security-group semantics for pod traffic, and some throughput from encapsulation plus a reduced effective MTU.
• Monitor allocation, not use: addresses allocated as a share of the network, and allocated-versus-used per node, as standing capacity metrics with a threshold at 70%.

Industry example

The constraint is why flat, routable pod networking - every pod holding a real network address - is both popular and hard to scale. The AWS VPC CNI's published guidance walks through exactly this arithmetic, and prefix delegation exists as a feature because the naive allocation wastes most of a private range. The wider version of the problem is older: estates that standardised on 10.0.0.0/8 in different business units, then merged or peered, and discovered overlapping ranges that no routing can reconcile. Translation layers between overlapping networks are a permanent tax paid for an address decision made once, quickly, years earlier.

Failure scenarios

• Autoscaling fails silently during a traffic peak, with pods pending and no capacity alert firing.
• A secondary range chosen for convenience collides with a partner's network at the first VPN or peering, and the fix is translation for ever.
• Subnets sized evenly across zones, so one zone exhausts while another has space and the scheduler cannot use it.
• Address reuse races. Addresses released by deleted pods are not immediately reusable in some CNI implementations, so a high-churn workload exhausts a prefix that looks half empty.
• An overlay introduced mid-life without MTU alignment, producing path MTU discovery failures that appear as large requests hanging while small ones succeed.

Trade-offs

When not to use it

The planning exercise is worth skipping below roughly 30 nodes: take the defaults, keep a spare /20 in the estate's plan and move on. Prefer flat routable addressing while it fits, because it is the model where security groups, flow logs and direct reachability behave as everyone expects, and an overlay adopted early is complexity bought against a constraint that may never arrive. The honest trigger for changing is not node count but estate scarcity - a second business unit needing space in the same private range, or a merger bringing overlapping networks.

Interview question

Q: Your cluster is in a /16 with a /24 per node and the roadmap says 400 nodes. Walk me through the capacity analysis and the options, with their costs.

What a strong answer covers: computing the prefix limit rather than the address limit and getting 256 nodes; discounting it further for subnets the rest of the estate needs; measuring real pod density before choosing; the four options with costs (right-size the prefix, prefix delegation, secondary CIDR with the collision caveat, overlay with the MTU and reachability caveats); the standing metrics that would give warning; and noting that the failure mode is pending pods during a scale-out with no gradual signal.

Quick check

Quiz: Why does a /16 with a /24 per node cap a cluster at 256 nodes? Because the binding resource is the number of /24 prefixes in the /16, not the 65,536 addresses it contains.

Flashcard: What does prefix delegation change? Allocation tracks actual pod density by handing out small prefixes on demand, cutting per-node waste from hundreds of addresses to a handful.