An observability platform's cloud bill shows an unexpectedly large data-processing charge from NAT gateways. What is happening, and what are the architectural fixes?

What is happening

NAT gateways charge for every gigabyte processed, in addition to an hourly rate. For a platform whose workload is moving very large volumes of telemetry, that per-gigabyte charge applied to traffic that never needed to traverse a NAT gateway at all becomes a significant line item.

The traffic is usually one of:

• Object storage access from private subnets. Instances writing to or reading from cloud object storage route through the NAT gateway to reach a public endpoint. This is frequently the largest single contributor, and it is entirely avoidable.
• Other cloud service API calls — queues, key management, container registries, monitoring endpoints — all reached over the public endpoint from private subnets.
• Container image pulls, which at scale and with large images are substantial.
• Package and dependency downloads during build or instance startup.
• Third-party integrations and outbound telemetry forwarding.

The pattern: traffic to services that live inside the cloud provider's own network is being routed out through a metered gateway and back in.

The fixes

1. Gateway endpoints for object storage. Traffic to object storage routes within the provider's network, bypassing the NAT gateway entirely, and is typically free. This single change usually removes the largest share of the cost, and it is a routing-table change rather than an architectural one.

2. Private endpoints for other cloud services. These have their own hourly and per-gigabyte cost, so the comparison is per-service: worthwhile for high-volume services, not for low-volume ones. Doing the arithmetic per service rather than adopting a blanket policy is the correct approach.

3. A registry cache or pull-through cache in the private network, so image pulls happen once from outside rather than once per instance.

4. Consolidate NAT gateways deliberately. One per availability zone is standard for resilience, and cross-zone traffic to reach a NAT gateway incurs its own transfer charge. Both the hourly cost and the cross-zone cost need to be considered together.

5. Measure before optimising. Flow logs tell you which destinations and which workloads dominate. Most teams guess wrong about this, and the distribution is usually far more concentrated than expected — a handful of destinations accounting for the large majority of bytes.

The general lesson

Network costs are architectural costs, and they are invisible in code review. A design that routes high-volume traffic through a metered path is not slower or less reliable, so nothing surfaces it except the bill — arriving a month later, aggregated, attributed to no team.

This is a strong argument for cost attribution and for including expected data-flow volumes in architecture review. The specific question worth asking of any design is: which of these arrows carries the most bytes, and what does that path cost per gigabyte?