Evidence ledger
34 sources
Checked 30 Aug 2026
Evidence ledger
One row per claim in Zone-local traffic and the cross-zone bill: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic. How production systems pay for the availability they buy by spreading across availability zones, and what breaks when they route the cross-zone bill away.
Hunt date. 2026-08-30. Every URL below was fetched during that session. Checked column is the date the link resolved.
One row per claim. Where an artefact supports several claims it appears several times. The right-hand column is copied text or a copied figure, never a paraphrase.
| # | Org | Title | Tier | Published | Checked | URL | Claim taken from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | AWS | Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region | postmortem | 2025-10 | 2026-08-30 | https://aws.amazon.com/message/101925 | The largest recent AWS outage was regional, not zonal; multi-AZ spreading bought nothing against it | "a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service's regional endpoint" |
| 2 | AWS | Summary of the Amazon DynamoDB Service Disruption (US-EAST-1) | postmortem | 2025-10 | 2026-08-30 | https://aws.amazon.com/message/101925 | Blast radius and duration of the October 2025 regional event | DynamoDB errors 11:48 PM PDT 19 Oct to 2:40 AM PDT 20 Oct; NLB connection errors 5:30 AM to 2:09 PM 20 Oct; cascading impact through 2:20 PM 20 Oct |
| 3 | Slack | Slack's Migration to a Cellular Architecture | postmortem | 2023-08-22 | 2026-08-30 | https://slack.engineering/slacks-migration-to-a-cellular-architecture/ | A degraded inter-AZ link, not a dead one, produced user-visible errors in every zone | "a network link that connects one availability zone with several other availability zones containing Slack servers experienced intermittent faults, causing slowness and degraded connections between Slack servers" |
| 4 | Slack | Slack's Migration to a Cellular Architecture | postmortem | 2023-08-22 | 2026-08-30 | https://slack.engineering/slacks-migration-to-a-cellular-architecture/ | RPC fan-out is what converts one degraded zone into a whole-service failure | "A single Slack API request from a user may fan out into hundreds of RPCs to service backends, each of which must complete to return a correct response." |
| 5 | Slack | Slack's Migration to a Cellular Architecture | postmortem | 2023-08-22 | 2026-08-30 | https://slack.engineering/slacks-migration-to-a-cellular-architecture/ | The structural fix is siloing plus a drain, with a stated time budget | services "may be said to be siloed if they only receive traffic from within their AZ and only send traffic upstream to servers in their AZ"; target is to remove "as much traffic as possible from an AZ within 5 minutes", shifted with 1% granularity |
| 6 | Broadcom | Availability Zone (use1-az4) Impacted | postmortem | 2026-05-08 | 2026-08-30 | https://status.broadcom.com/notices/xxpqglmvmv4dab7q-availability-zone-use1-az4-impacted | A single-AZ impairment removed control-plane operations for a downstream vendor for roughly two days | Began 02:02 UTC 8 May 2026, resolved 00:49 UTC 10 May 2026; impacts "No new host provisioning", "No new SDDC deployments", "Failure of auto-remediation for host failures" |
| 7 | Network World (Gyana Swain) | AWS hit by US-East-1 outage after data center thermal event | postmortem | 2026-05-11 | 2026-08-30 | https://www.networkworld.com/article/4168878/aws-hit-by-us-east-1-outage-after-data-center-thermal-event.html | Zonal failure is still a live event class as of three months ago, and it hit customers who had not spread | AWS statement quoted: "EC2 instances and EBS volumes hosted on impacted hardware are affected by the loss of power during the thermal event"; "servers automatically shut down when the temperatures exceeded the operating thresholds in order to protect the hardware". Cooling restored by 1:50 PM PDT Friday 8 May, thermal event late Thursday 7 May |
| 8 | Envoy | Issue #28419: Zone-aware load balancing incorrectly handles mismatched local and upstream localities | source | 2023-07-16 | 2026-08-30 | https://github.com/envoyproxy/envoy/issues/28419 | The affinity mechanism itself can skew load fivefold, and can be silently on for some clients and off for others | "zone-aware routing is enabled for some envoys in the local cluster, and disabled for others"; "zone D will receive 5x as much traffic from the two local envoys in zone A". Closed, fixed via PR #28970 |
| 9 | Envoy | Issue #43665: Proposal: ORCA-driven locality routing for zone-aware load balancing | source | 2026-02-26 | 2026-08-30 | https://github.com/envoyproxy/envoy/issues/43665 | Zone-aware routing balances on endpoint counts, not on load, and the project acknowledges it | "It has no visibility into actual backend load."; "least-request helps within a zone but can't redistribute across zones". Closed, linked PR #45725 |
| 10 | Kubernetes | Issue #122437: Topology Aware Routing: Unable to allocate minimum required endpoints to each zone without exceeding overload threshold | source | 2023-12-21 | 2026-08-30 | https://github.com/kubernetes/kubernetes/issues/122437 | Routine scaling silently disables the optimisation | Error: "Unable to allocate minimum required endpoints to each zone without exceeding overload threshold (11 endpoints, 3 zones)". 9 replicas at [3,3,3] gets hints; 11 replicas at [4,4,3] loses them. Closed. Reported on v1.27.6, AWS |
| 11 | Kubernetes | Issue #121516: Topology Aware Routing's OverloadThreshold should be configurable per Service | source | 2023-10-25 | 2026-08-30 | https://github.com/kubernetes/kubernetes/issues/121516 | A three-pod, three-zone service cannot use the feature at all under the hardcoded threshold | Reporter runs three Loki pods across three AZs; the controller refuses, saying it cannot "allocate minimum required endpoints to each zone without exceeding overload threshold". Closed as completed 2024-04-30 |
| 12 | Kubernetes | PR #103924: [WIP] Traffic Policy and Topology Aware Routing should not be exclusive | source | 2021-07-26 | 2026-08-30 | https://github.com/kubernetes/kubernetes/pull/103924 | The attempt to combine node-local traffic policy with zone-aware routing was abandoned unmerged | State closed, merged_at null, closed 2022-01-14 after five and a half months. Release note: "Traffic Policy and Topology Aware Routing should not be exclusive" |
| 13 | Apache Kafka | KAFKA-19387: Support rack awareness in Range and Uniform assignors | source | 2025-06-07 | 2026-08-30 | https://issues.apache.org/jira/browse/KAFKA-19387 | The newer consumer group protocol does not yet carry the rack-aware assignment that the cost optimisation depends on | Status "In Progress", no fix version, priority Major. Four sub-tasks covering homogeneous and heterogeneous assignment in both assignors |
| 14 | Apache Kafka | KIP-392: Allow consumers to fetch from closest replica | adr | 2019 | 2026-08-30 | https://cwiki.apache.org/confluence/display/KAFKA/KIP-392:+Allow+consumers+to+fetch+from+closest+replica | Follower fetching exists because of inter-datacenter traffic cost, with availability zones treated as datacenters | "consumers are limited to fetching only from the leader, so there is no easy way to leverage locality in order to reduce expensive cross-dc traffic"; "a common deployment is within an AWS region in which each availability zone is treated as a datacenter" |
| 15 | Apache Kafka | KIP-392: Allow consumers to fetch from closest replica | adr | 2019 | 2026-08-30 | https://cwiki.apache.org/confluence/display/KAFKA/KIP-392:+Allow+consumers+to+fetch+from+closest+replica | The correctness price of reading from a follower, and the rejected alternative | Follower fetching "may have higher latency" from watermark propagation and introduces "the possibility of spurious out of range errors if a fetch is received by an out-of-sync replica". Consumer-side replica selection rejected because "it is easier to reason about the system in a multi-tenant environment when replica selection logic could be controlled by the broker" |
| 16 | Apache Kafka | KIP-881: Rack-aware Partition Assignment for Kafka Consumers | adr | 2022 | 2026-08-30 | https://cwiki.apache.org/confluence/spaces/KAFKA/pages/231118120/KIP-881+Rack-aware+Partition+Assignment+for+Kafka+Consumers | Rack-aware assignment reaches the same outcome as follower fetching by a different route | Consumers "benefit from locality and avoid expensive cross-AZ traffic by configuring client.rack"; KIP-881 propagates client rack to the assignor so it can align racks by assigning the right broker in the same AZ |
| 17 | Apache Kafka | KIP-1183: Unified Shared Storage | adr | 2025-05-16 | 2026-08-30 | https://cwiki.apache.org/confluence/display/KAFKA/KIP-1183%3A+Unified+Shared+Storage | Kafka is being redesigned around this line item, and there are competing proposals | Status "Under Discussion". "The community has made significant efforts to reduce cross-AZ traffic for producers and consumers"; on shared storage "data replication is unnecessary" |
| 18 | Kubernetes | KEP-2433: Topology Aware Hints | adr | 2021-02 | 2026-08-30 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/2433-topology-aware-hints/README.md | The safeguard thresholds are hardcoded, and the KEP names the overload risk in advance | Overload threshold 30% maximum acceptable before redistribution; hints only added when initial expected overload is below 20%. "In a scenario where all traffic originates from a single zone there is a chance that endpoints in that zone will be overloaded while endpoints in other zones receive little to no traffic." Alpha 1.21, beta 1.23, GA 1.33 with only the hints field graduating |
| 19 | Kubernetes | KEP-3015: PreferSameZone and PreferSameNode Traffic Distribution | adr | 2025 | 2026-08-30 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/3015-prefer-same-node/README.md | The project renamed its own option because the first name misdescribed what it did | "any attempt to change the semantics of PreferClose would inevitably have either too many false positives or too many false negatives. Thus, this KEP deprecates PreferClose and replaces it with PreferSameZone as a new name for the old behavior." |
| 20 | Kubernetes | Topology Aware Routing (documentation) | vendor | 2026 | 2026-08-30 | https://kubernetes.io/docs/concepts/services-networking/topology-aware-routing/ | Five documented conditions silently turn the optimisation off | "If there are less endpoints than zones in a cluster, the controller will not assign any hints."; on a partial-hint state, "Filtering endpoints for a Service in this state would be dangerous so the kube-proxy falls back to using all endpoints." Also incompatible with internalTrafficPolicy: Local; "Does not work well when large proportion of traffic originates from a subset of zones" |
| 21 | Envoy | Zone aware routing (documentation) | vendor | 2026 | 2026-08-30 | https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/zone_aware | Zone affinity in Envoy is conditional on health, and only applies at the highest priority | Preconditions include "Both originating and upstream cluster are not in panic mode"; "zone aware routing is currently only supported for P=0". Default panic threshold is 50% |
| 22 | AWS (Becky Weiss, Mike Furr) | Static stability using Availability Zones, Amazon Builders' Library | vendor | 2019 | 2026-08-30 | https://d1.awsstatic.com/builderslibrary/pdfs/static-stability-using-availability-zones.pdf | AWS derives zone-local traffic as a reliability pattern, with the arithmetic, and treats cost as the side benefit | "if a request is sent through the system, then with some simplifying assumptions, the chance of the request avoiding the impaired Availability Zone is ⅔ * ⅔ = 4/9 ... If N services are a part of this call path, then these numbers generalize to (⅔)^N for N regional services versus remaining constant at ⅔ for N zonal services." And: this pattern "not only helps keep impairments isolated to an Availability Zone but also has favorable network traffic cost characteristics in AWS" |
| 23 | AWS (Becky Weiss, Mike Furr) | Static stability using Availability Zones | vendor | 2019 | 2026-08-30 | https://d1.awsstatic.com/builderslibrary/pdfs/static-stability-using-availability-zones.pdf | The capacity price of three zones, stated exactly | "when we use three Availability Zones, we overprovision by 50 percent. Put another way, we overprovision such that each Availability Zone is operating at only 66 percent of the level for which we have load-tested it." |
| 24 | AWS (Becky Weiss, Mike Furr) | Static stability using Availability Zones | vendor | 2019 | 2026-08-30 | https://d1.awsstatic.com/builderslibrary/pdfs/static-stability-using-availability-zones.pdf | Scaling in response to a zone impairment is named as the weaker design | "This approach is less effective because it relies on reacting to impairments as they happen, rather than being prepared for those impairments before they happen. In other words, it lacks static stability." |
| 25 | Microsoft Research (Huang, Guo, Zhou, Lorch, Dang, Chintalapati, Yao) | Gray Failure: The Achilles' Heel of Cloud-Scale Systems, HotOS '17 | paper | 2017-05 | 2026-08-30 | https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf | Failure detectors and applications disagree, which is why a degraded zone is not drained | "a key feature of gray failure is differential observability: that the system's failure detectors may not notice problems even when applications are afflicted by them"; "gray failure is behind most cloud incidents" |
| 26 | Microsoft Research | Gray Failure: The Achilles' Heel of Cloud-Scale Systems | paper | 2017-05 | 2026-08-30 | https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf | Adding redundancy across zones can lower availability under gray failure | Section 2.1 is titled "High redundancy hurts". "we sometimes see cases where increasing redundancy actually lowers availability" |
| 27 | Uber and UBC (Bansal, Chabbi, Bøgh et al.) | Uber's Failover Architecture: Reconciling Reliability and Efficiency in Hyperscale Microservice Infrastructure, arXiv:2603.07345 | paper | 2026-03-07 | 2026-08-30 | https://arxiv.org/pdf/2603.07345 | The event that justifies standing redundancy is measurably rare, and measuring it changes the spend | "full-peak regional failovers average fewer than 20 hours annually, just 0.23% of the time"; "UFA reduces steady-state provisioning from 2x to 1.3x, raising utilization from 20% to 30% while sustaining 99.97% availability" |
| 28 | Uber and UBC | Uber's Failover Architecture, arXiv:2603.07345 | paper | 2026-03-07 | 2026-08-30 | https://arxiv.org/pdf/2603.07345 | The absolute size of the redundancy bill at hyperscale | "eliminated over one million CPU cores from a baseline of about four million cores"; "over 4,000 unsafe dependencies" hardened; over 6,000 microservices in over 16,000 environments |
| 29 | Buoyant (William Morgan) | The trouble with Topology Aware Routing: Sacrificing reliability in the name of cost savings | blog | 2024-06-05 | 2026-08-30 | https://www.buoyant.io/blog/the-trouble-with-topology-aware-routing-sacrificing-reliability-to-avoid-cross-zone-traffic | Zone-local routing without a spill path zeroes out the zone it is protecting | "TAR prevents pods in the failing zone from ever being able to reach pods in the other zones to compensate"; "we drop to 0% success rate in that zone, as its orders pods are unable to reach any other pod, despite the fact that there are healthy pods elsewhere in the cluster!" |
| 30 | Buoyant (William Morgan) | The trouble with Topology Aware Routing | blog | 2024-06-05 | 2026-08-30 | https://www.buoyant.io/blog/the-trouble-with-topology-aware-routing-sacrificing-reliability-to-avoid-cross-zone-traffic | Published unit prices and an annualised figure for a sustained rate | AWS \(20/terabyte cross-AZ; Google Cloud \)10.24/terabyte; at 1 GB/s average traffic on AWS, $410,000 annually |
| 31 | Grab (Fabrice Harbulot, Quang Minh Tran) | Zero traffic cost for Kafka consumers | blog | 2023-07-07 | 2026-08-30 | https://engineering.grab.com/zero-traffic-cost | Measured outcome and measured price of enabling follower fetching in production | 25% reduction in cross-AZ traffic after three months; cross-AZ traffic had been "half of the total cost" of the Kafka platform; up to 500 ms increase in end-to-end p99 latency; visible consumer load skew after 1 March 2023 requiring extra brokers in overloaded zones |
| 32 | Grafana Labs (Zhehao Zhou) | The next generation of Grafana Mimir: Inside Mimir's redesigned architecture for increased reliability | blog | 2025-02-21 | 2026-08-30 | https://grafana.com/blog/the-next-generation-of-grafana-mimir-inside-mimirs-redesigned-architecture-for-increased-reliability/ | A production architecture was redesigned because the cross-zone charge made the reliable shape unaffordable | "the cross-availability zone data transfer charges it incurred made this cost-prohibitive to do on a large scale"; the replacement "has no inter-AZ networking costs and no disks to manage" |
| 33 | Jack Vanlightly | A Fork in the Road: Deciding Kafka's Diskless Future | blog | 2025-10-22 | 2026-08-30 | https://jack-vanlightly.com/blog/2025/10/22/a-fork-in-the-road-deciding-kafkas-diskless-future | Three competing Kafka designs were live at once over this one cost, and a practitioner warns against reading it as only a cost problem | Quotes Luke Chen on "three KIPs simultaneously addressing the same challenge of high replication costs when running Kafka across multiple cloud availability zones"; and "Cross-AZ charges are typically what are on people's minds, but it's a mistake to think of S3 simply as a cheaper disk or a networking cheat" |
| 34 | InfoQ (Eran Stiller), on DoorDash | DoorDash Uses Service Mesh to Manage Data Transfer and Reduce Cloud Spend | casestudy | 2024-01-23 | 2026-08-30 | https://www.infoq.com/news/2024/01/doordash-service-mesh | Zone-aware routing at mesh level produced a saving large enough for the cloud provider to ask whether something had broken | DoorDash engineers Hochuen Wong and Levon Stepanian: "these actions made such a material dent in DoorDash's data transfer costs [..] that it caused our cloud provider to reach out to us asking whether we were experiencing a production-related incident." Also: "As the number of hops increases in microservice call graphs, the likelihood of data being transmitted across AZs grows." |
| 35 | eBPF Foundation, on Polar Signals | Polar Signals Uses eBPF to Monitor Internal Cross-Zone Network Traffic on Kubernetes, Reducing These Operating Costs by 50% | casestudy | 2025-01-15 | 2026-08-30 | https://ebpf.foundation/case-study-polar-signals-uses-ebpf-to-monitor-internal-cross-zone-network-traffic-on-kubernetes-reducing-these-operating-costs-by-50/ | Attribution had to be built before optimisation was possible, and the share of spend was about half | "Cross-zone traffic accounted for nearly half of their cloud expenses" (mid-2024); costs reduced by 50%; the tool built is kubezonnet, using eBPF to trace packets and expose Prometheus metrics and flow logs |
| 36 | Confluent (Addison Huddy) | Uncovering Kafka's Hidden Infrastructure Costs | vendor | 2023-04-20 | 2026-08-30 | https://www.confluent.io/blog/understanding-and-optimizing-your-kafka-costs-part-1-infrastructure/ | Where cross-zone bytes come from in a replicated log, and the share of the bill | "networking likely represents over 50% of your Kafka infrastructure costs"; producers write to leaders in other zones about ⅔ of the time, consumers read from remote leaders about ⅔ of the time, replication at factor 3 adds 2x the ingress; cross-AZ transfer priced at $0.02 per GB |
| 37 | Confluent | Confluent Cloud Freight Clusters in GA | vendor | 2025-02-03 | 2026-08-30 | https://www.confluent.io/blog/freight-clusters-are-generally-available/ | The commercial answer trades latency for the whole cross-zone line item | Freight can "completely remove cross-zone traffic costs"; latency moves from "sub-100ms" in standard clusters to "up to a second or two with Freight"; claim of "90% lower infrastructure costs" |
| 38 | InfoQ (Renato Losio) | Microsoft Drops Data Transfer Fees between Availability Zones | blog | 2024-06-08 | 2026-08-30 | https://www.infoq.com/news/2024/06/azure-az-transfer-fees/ | One of the three major clouds deleted the charge, and said why | Microsoft: "We aim to support customers in adopting the best practice architecture for cloud deployment. Removing cross-AZ transfer costs encourages the use of highly available and fault-tolerant system architectures without additional fees." Comparison given: AWS $0.01 per GB per direction; Google Cloud similar; Azure now free |
| 39 | Microsoft Azure | Bandwidth pricing | vendor | 2026 | 2026-08-30 | https://azure.microsoft.com/en-us/pricing/details/bandwidth/ | Confirms the current Azure price is zero | Pricing table: "Data transfer within same Availability Zone — Free"; FAQ confirms no charge for "data transfer between Azure services located within the same region" |
| 40 | AWS (Bingjiao Yu, Niall Thomson, Xiangyan Wang) | Exploring the effect of Topology Aware Hints on network traffic in Amazon EKS | vendor | 2023-05-16 | 2026-08-30 | https://aws.amazon.com/blogs/containers/exploring-the-effect-of-topology-aware-hints-on-network-traffic-in-amazon-elastic-kubernetes-service/ | AWS's own demonstration is qualitative, not quantitative, and names the balance precondition | Before: "Each UI pod sends traffic to all three catalog instances across all of the AZs". After: "UI pod always talks to catalog pod resides on the same AZ. No inter-AZ data traffic cost is incurred". Warns that horizontal pod autoscaling may disrupt zone balance during scale-in |
| 41 | Confluent / Kafka Summit SF (Uday Sagar Siramshetty, SignalFx) | Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka | talk | 2019 | 2026-08-30 | https://www.confluent.io/kafka-summit-san-francisco-2019/achieving-a-50-reduction-in-cross-az-network-costs-from-kafka/ | The problem and the 50% figure predate every mechanism now used to solve it | Talk abstract: cloud providers "allow free data transfers within an Availability Zone but bill users for cross-AZ data movement"; covers architectural choices to produce and consume within a single AZ, and compression during replication |
| 42 | KubeFM (William Morgan) | Topology-aware routing: balancing cost savings and reliability | talk | 2025-01-21 | 2026-08-30 | https://kube.fm/topology-aware-routing-william | The trade-off is being argued in public, in the practitioner channel, a year after the feature shipped broadly | Episode with William Morgan, CEO of Buoyant, dated 21 January 2025, on topology-aware routing trade-offs and cost management in Kubernetes |
| 43 | WarpStream | How WarpStream Powers Grafana Labs' Redesigned Architecture | vendor | 2025-02-23 | 2026-08-30 | https://www.warpstream.com/blog/how-warpstream-powers-grafana-labs-redesigned-architecture | The supplier's corroboration of Grafana's choice, with the throughput it was tested at | "no inter-AZ networking costs and no disks to manage"; load tested at "1.5 GiB/s (compressed) through a single WarpStream cluster" consumed "with 4x fan-out for an aggregate compressed throughput of 7.5 GiB/s" |
Tier mix
| Tier | Rows | Distinct artefacts |
|---|---|---|
| postmortem | 7 | 4 |
| source | 6 | 6 |
| adr | 6 | 6 |
| paper | 4 | 2 |
| casestudy | 2 | 2 |
| blog | 6 | 5 |
| talk | 2 | 2 |
| vendor | 10 | 9 |
34 distinct artefacts across 22 hosts. Vendor material is 9 of 34, about a quarter, and every vendor claim in the page is labelled as a claim.
Where the record runs out
Recorded here so the page can say it plainly.
- No public postmortem attributes an incident to zone-affinity routing itself. Every failure mode in section 4 that involves the affinity mechanism comes from an issue tracker, a vendor demonstration, or a project's own documented safeguards. Nobody has published "our zone-local routing caused this outage" with a timeline. Two readings: the safeguards work, or the failure is attributed to something else in the write-up. Both readings argue for measuring the spill path yourself.
- No independent measurement of the Confluent Freight or WarpStream cost claims. The 90% figure and the "zero inter-AZ cost" claim are vendor statements. Grafana Labs corroborates the direction by adopting one of them, but publishes no cost delta.
- No public account of the operational cost of keeping zones balanced. Grab needed extra brokers in overloaded zones and Kubernetes disables hints on imbalance, so balance clearly has a price, but nobody has published what it is.
- The Buoyant annualised figure does not reconcile with the unit price. 1 GB/s sustained is about 31.5 PB per year; at AWS's \(0.01/GB that is \)315,360 one direction and \(630,720 billed both ways. The published \)410,000 sits between the two and the working is not shown. Treat the unit price as the primary figure and derive your own.