Conntrack Table Ceiling

Any device that translates addresses or filters statefully keeps a row per flow. The number of rows in steady state is not the number of active connections users would describe; it is:

entries ≈ new connections per second × entry lifetime

For short-lived connections the lifetime is dominated by the post-close hold - commonly about 120 seconds - not by how long the connection was useful. So 12,000 new connections per second implies roughly 1.44 million entries, which at about 300 bytes each is around 430 MB of state for traffic that is only a few hundred megabits per second.

The ceiling is reached while CPU, memory and bandwidth all look healthy, and it reports itself as intermittent connection failures with no error on either endpoint, which is why investigations start by looking for capacity.

Why it matters

It is a capacity dimension absent from most capacity models. Teams size egress on bandwidth and instances on CPU, and the thing that breaks is a table whose default they never saw.

The defaults make it worse. Many Linux builds ship a connection-tracking maximum in the low hundreds of thousands, so the example above exceeds the default by roughly five times; managed NAT services publish their own per-gateway ceilings, which are generous but finite and shared across everything behind the gateway.

And the companion limit is harder. The port range, not the table, is usually the real wall. A translated flow is identified by public source address, source port, destination address and destination port. For one public address to one destination, only the source port varies - roughly 28,000 usable - and each port is unavailable until its hold expires:

28,000 ports / 120 s ≈ 233 new connections per second, per destination, per public address

Sustaining 12,000 connections per second to a single destination therefore needs about 52 public addresses, which is a procurement fact disguised as a networking detail.

Implementation patterns

• Compute both numbers at design time: entries from rate times lifetime, and connections per second per destination from port range divided by hold.
• Export the table-usage metric and alert at about 70%. On Linux this is the connection count against the configured maximum; managed gateways expose an equivalent. Without this metric the failure is undiagnosable from the application side.
• Reuse connections rather than creating them. A pooled client holding 1,000 concurrent connections needs 1,000 entries where per-request connections need 1.44 million - the largest lever, and an application change.
• Shorten the post-close hold where the protocol and risk tolerance allow, which cuts the table proportionally and doubles the per-address connection rate.
• Add public source addresses when traffic concentrates on few destinations, since each one multiplies the tuple space.
• Route in-cloud traffic through private endpoints, so object-storage and managed-service traffic consumes no translation state at all. This commonly removes most of the volume.
• Spread flows across gateways with distinct address pools when a gateway's published ceiling is genuinely the constraint.

Industry example

The pattern is routine in container platforms and crawling fleets, where thousands of short-lived outbound connections per second pass through shared egress. Cloud providers have published per-gateway connection ceilings since managed NAT services appeared in 2017 to 2018, and recommend more addresses or private endpoints as the remedies; Linux kernel documentation for the connection-tracking subsystem has long warned that the default maximum is derived from available memory and is unlikely to suit high-rate workloads. The recurring incident shape is a batch job scaled by a factor of ten, intermittent failures with no resource alert, and days spent on the application before someone reads the table metric.

Failure scenarios

• Table full under load. New flows are dropped or entries evicted, producing random connection failures while every dashboard is green.
• Port exhaustion against one destination - a fleet talking to one managed database or API endpoint, where the ceiling is about 230 connections per second per address and has nothing to do with the destination.
• Scaling out the wrong thing. More application instances behind the same gateway multiply the connection rate and reach the ceiling sooner.
• Adding a second gateway with the same address pool, which does not increase the tuple space.
• Eviction of long-lived flows. Under pressure some devices evict older entries, so a long-running replication link dies first - the opposite of the intended behaviour.
• Invisible retries. A client retrying a failed connection raises the arrival rate, growing the table further, which is a self-reinforcing loop.

Trade-offs

When not to use it

This is not a useful lens for a workload with few long-lived connections - a mesh of persistent gRPC channels, a pooled database fleet. There the entry count is small and the interesting limits are memory per connection and file descriptors. It is also the wrong model for inbound traffic terminated by a load balancer you do not translate through, and for stateless filtering, which keeps no rows at all.

And where egress is mostly to a cloud provider's own services, do the private-endpoint work first rather than sizing a table: the answer is to stop generating the state, not to hold more of it.

Interview question

Q: A batch fleet is scaled from 20 to 200 workers and starts seeing intermittent connection failures to one external API. CPU, memory and bandwidth are all comfortable and the API provider reports no errors. What is your hypothesis, how do you confirm it, and what do you change?

What a strong answer covers: entries as rate times lifetime, and the port ceiling of roughly 230 connections per second per destination per address · why concentration on one destination makes the port ceiling bind first · confirming with the table-usage metric rather than guessing · pooling as the largest lever, then a shorter hold, then more addresses · private endpoints for in-cloud destinations · and the retry loop that makes the symptom intermittent.

Quick check

Quiz: Why does adding a second NAT gateway sometimes fail to fix port exhaustion? Because the ceiling is set by the tuple space of the public source addresses, so a second gateway sharing the same address pool adds no new identities - more addresses, pooling or a shorter hold are the real levers.

Flashcard: Roughly how many tracked entries does 12000 new connections per second with a 120-second hold imply, and what is the second ceiling? - About 1.44 million entries, roughly five times the default maximum on many Linux builds; the second and usually tighter ceiling is about 233 connections per second per destination per public address.