Invisible Resource Exhaustion

Some limits fail without any signal in the place that experiences them. NAT gateway source ports, connection tracking table entries, ephemeral port ranges, per-account cloud API quotas, and load balancer connection limits all share a property: when they are exhausted, the application logs a connection failure and every local metric is normal.

CPU is fine. Memory is fine. The dependency is up. The error is intermittent and the pattern makes no sense.

Why it matters

These are among the hardest production problems to diagnose because the diagnostic instinct — look at the resource that is exhausted — fails, since nothing in the application's view is exhausted. Teams typically spend significant time investigating the application and the dependency before considering the path between them.

Implementation patterns

• Instrument the shared resources explicitly: NAT port allocation and error counts, connection tracking table utilisation, quota consumption against limits, load balancer connection counts. Nothing else in the stack will report them, so they must be deliberately collected.
• Alert on utilisation, not on failure. These resources fail abruptly at 100%, so an alert on errors fires after the outage has begun. An alert at 70% utilisation gives time.
• Reduce demand at the source: connection pooling and reuse, since the underlying cause is frequently a client opening a connection per request.
• Remove high-volume destinations from the shared path — private or gateway endpoints for object storage, direct peering for major partners. A small number of destinations usually dominates, so this is often the largest single reduction.
• Partition the resource, with egress distributed across gateways per zone or workload class, so exhaustion is contained rather than total.
• Record the diagnosis. Because these problems recur and are unintuitive, a written note connecting the symptom to the cause saves the next engineer several hours.

Industry example

A platform such as BrowserStack running thousands of concurrent isolated browser sessions, each loading dozens of external resources, generates an extraordinary number of simultaneous outbound connections. Port exhaustion is the natural consequence, made worse by connections in TIME_WAIT holding their allocation after closing, so the effective ceiling arrives well below the theoretical one.

The same class affects any high-concurrency egress workload: crawlers, aggregators fanning out to hundreds of providers, CI systems fetching dependencies, and inference platforms calling external tools.

Failure scenarios

• Intermittent connection failures investigated as an application bug for days.
• Alerting on errors rather than on utilisation, so warning arrives with the outage.
• Adding capacity to the application, which increases concurrency and makes it worse.
• A single shared gateway for all egress, so one workload's behaviour affects everything.
• The limit rediscovered annually because the diagnosis was never written down.

Trade-offs

Instrumenting every shared limit is work with no visible payoff until the day it matters, and it competes with features. The mitigation is to instrument the ones with known exhaustion modes — egress, quotas, connection tracking — rather than attempting completeness.

Partitioning the resource costs money (more gateways, more addresses) and configuration complexity, in exchange for containment. For a high-concurrency egress workload that is straightforwardly worth it; for a low-volume service it is over-engineering.

Interview question

"Your service intermittently fails to reach an external API. Latency is normal, CPU is low, the API reports no problem, and the failures come in bursts. Give me five candidate causes and the single measurement that would distinguish between them."