A batch job's networking costs are three times its compute cost. It reads several terabytes from object storage. What is happening?
Show the full answer Hide the answer
What is being tested
Knowledge of a specific, extremely common, and entirely avoidable cloud cost trap — and the general habit of asking which components a high-volume flow traverses.
The mechanism
The compute runs in a private subnet, which by definition has no direct internet route. To reach any service by its public endpoint — including the provider's own object storage — traffic must exit through a NAT gateway.
NAT gateways bill both per hour and per gigabyte processed. That per-gigabyte charge applies to all traffic, including traffic destined for a service inside the same provider that never needed to traverse the public internet at all.
At several terabytes, the processing charge alone dwarfs the compute cost of the job. Nothing is broken; the traffic is simply taking an expensive path for no reason.
The fix
A private endpoint (variously called a VPC endpoint, private link, or service endpoint depending on the provider) creates a route from the subnet directly to the provider's service, bypassing the NAT gateway entirely. Traffic no longer incurs NAT processing charges.
This is typically a small infrastructure change, carries almost no risk, and routinely removes a large fraction of an unexplained networking bill. It is among the highest-return changes available in a cloud estate.
The related NAT issues worth knowing
- One NAT gateway per availability zone. A single shared gateway means cross-zone transfer charges on every outbound connection and a zone-level single point of failure for all outbound connectivity.
- Port exhaustion. A NAT gateway supports a finite number of simultaneous connections per destination. A workload making very many connections to one destination — a polling client, a chatty service — can exhaust them, producing intermittent connection failures that look like a downstream problem and are very hard to diagnose.
- Container image pulls through NAT on every scale-out event, which is both slow and charged. A registry pull-through cache or a private endpoint to the registry fixes it.
- Egress is unfiltered by default. NAT permits outbound to anywhere, which is the path exfiltration takes. If that matters, a proxy with an allowlist belongs in the path.
Why the other options are wrong
Object storage reads within the same region are generally not charged as egress — the charge here is NAT processing, which is why the private endpoint fixes it. Cross-region transfer would be a real cost but is a different diagnosis with a different fix. Instance sizing affects duration, not the per-gigabyte networking charge.
The transferable habit
For any high-volume data flow, ask: which billed components does this traverse, and does it need to? NAT gateways, load balancers, cross-zone hops and inter-region links are all metered, and a flow routed through them unnecessarily is a recurring charge that nobody notices because it appears under "networking" rather than next to the workload that causes it.