Zone-local traffic and the cross-zone bill, a practitioner field guide

The territory

The problem, without naming a technology: you split a system across three independent failure domains so that losing one does not lose the system, and you are then charged per byte for every crossing between them. The charge grows with traffic; the protection does not. So teams route around the charge.

$0.02

Per GB round trip for cross-AZ traffic on AWS, a cent in each direction

Confluent, 2023

>50%

Share of Kafka infrastructure cost that is networking, when spread across zones

Confluent, 2023

$0.00

Azure's cross-zone rate since 21 May 2024, for private and public IPs alike

InfoQ, 2024

0.23%

Of the year Uber spent in the full-peak failover its 2× capacity model paid for

Uber, arXiv, 2026

The third of those four numbers is the one that should stop a reader. Microsoft stopped charging for data transfer between availability zones on 21 May 2024, for both private and public IP addressing, and said why in plain terms: "Removing cross-AZ transfer costs encourages the use of highly available and fault-tolerant system architectures without additional fees." AWS and Google Cloud still charge. So an entire body of engineering work, an entire product category, and at least three competing redesigns of Apache Kafka exist to route around a line item that one of the three major clouds has already decided should not exist. That is not a physical cost being optimised. It is a pricing decision being optimised, and it can be repriced without warning.

The second thing worth noticing is who first wrote down the argument for keeping traffic inside a zone. It was not a cost-optimisation post. It was AWS's Amazon Builders' Library in 2019, in an article about static stability, and the argument is about survival odds, not dollars. If a zone is impaired and your call path crosses N regionally load-balanced services, the chance a request avoids the impaired zone is (2/3)^N. If every hop is zone-local, it stays at 2/3 no matter how deep the call graph goes. Becky Weiss and Mike Furr mention the money once, at the end, as an afterthought: this pattern "not only helps keep impairments isolated to an Availability Zone but also has favorable network traffic cost characteristics in AWS."

Seven years later the ordering has reversed. Kubernetes ships zone-local routing as a cost feature. The best-known critique of it, from Buoyant in June 2024, is titled "Sacrificing reliability in the name of cost savings". Both framings are supported by evidence and they contradict each other, which is the most interesting fact in this corpus. The resolution, which section 2 sets out, is that AWS and Kubernetes are not describing the same mechanism. AWS describes zone affinity applied to an entire call path that can be evacuated as a unit. Kubernetes describes zone affinity applied to one hop with no evacuation mechanism at all. The first is a reliability pattern. The second is a cost feature wearing its clothes.

The finding to carry into a design review

Zone-local routing is safe in proportion to how completely the call path is siloed and how fast the zone can be drained. Applied to one hop in a mesh with no drain, it removes cross-zone spend and cross-zone recovery in the same configuration change, and only one of those two shows up on a dashboard.

Who has solved this in production

Six organisations have published enough operational detail to reconstruct from, and they cluster into three postures. Silo and drain: Slack, and AWS's own foundational services such as NAT Gateway. Prefer and spill: DoorDash and Grab, who made the common path zone-local while keeping a health-driven escape route. Remove the crossing entirely: Grafana Labs, who redesigned Mimir onto object storage because, in Zhehao Zhou's words in February 2025, "the cross-availability zone data transfer charges it incurred made this cost-prohibitive to do on a large scale." Polar Signals sits earlier in the pipeline than any of them: they built an eBPF tool to find out where the bytes were going before changing anything, having discovered in mid-2024 that cross-zone traffic was nearly half their cloud bill.

Figure 1 · Where the cross-zone bytes come from

The charge is not one crossing, it is a crossing per hop, and a replicated log multiplies it again. Confluent's decomposition of a three-zone Kafka cluster is that producers reach a leader in another zone about two thirds of the time, replication at factor three adds twice the ingress, and consumers read a remote leader about two thirds of the time. Sources: Confluent, 2023, DoorDash via InfoQ, 2024.

Diagram source

Scope

This guide covers traffic between availability zones inside one cloud region: what it costs, the mechanisms used to keep it local, and how those mechanisms behave when a zone degrades. It deliberately does not cover cross-region traffic or multi-region disaster recovery, internet egress and CDN economics, quorum and replica placement for consensus latency, data residency and compliance constraints on locality, or on-premises rack topology. Cell-based partitioning as a general containment strategy is covered in the companion guide under Platform & Infrastructure; here it appears only where the cell boundary is a zone boundary.

How it is actually built

Across every system in this corpus, zone affinity is assembled from the same five layers. Four of them are widely implemented. The fifth is what separates the systems that survived a zone event from the ones that discovered they had a problem.

Nobody in the sources names this stack, because each organisation built its own slice and described it in its own vocabulary: Kafka calls it rack awareness, Kubernetes calls it topology-aware routing, Envoy calls it locality load balancing, Slack calls it siloing. Laying the accounts side by side, the same five responsibilities appear every time, and the difference between the implementations is which of the five they include. I am calling it the affinity stack; the sources call it four different things.

Figure 2 · The affinity stack, reconstructed across six systems

Layers 1 to 4 are present in every published implementation. Layer 5, the drain, appears only in Slack's account and in AWS's design of its own zonal services; Kubernetes Topology Aware Routing has no equivalent. Reconstructed from Slack, DoorDash, Grab and the Kubernetes documentation.

Diagram source

1 · Placement

Every mechanism above this layer assumes endpoints are spread proportionally to capacity. Kubernetes computes hints from allocatable CPU per zone and refuses to emit them when the spread will not work. Kafka uses broker.rack so replicas of a partition land in different zones.

Runs this way at: Grab, AWS EKS reference

2 · Discovery

The zone has to reach the thing making the routing decision. Kubernetes attaches hints to EndpointSlices; DoorDash modified its mesh control plane to put each node's availability zone into the endpoint discovery response; Kafka consumers set client.rack from the instance's AZ ID at startup.

Runs this way at: DoorDash, Grab

3 · Selection

The decision itself, and the layer where the implementations genuinely differ. Envoy computes a residual capacity split and refuses to do any of it in panic mode; kube-proxy filters the endpoint list; Kafka's RackAwareReplicaSelector hands the consumer a same-zone replica.

Runs this way at: Envoy, Kafka KIP-392

4 · Replication

Stateful systems cannot route around durability. Either you pay to replicate across zones, or you push the durability into a store that is already regional. Grafana Labs took the second route for Mimir; Confluent sells it as Freight; Kafka is arguing about which version of it to adopt.

Runs this way at: Grafana Labs, Kafka KIP-1183

5 · Drain

An operator command that removes all traffic from one zone in minutes. Slack targets "as much traffic as possible from an AZ within 5 minutes", shifted at 1% granularity so in-flight requests complete. Nothing in Kubernetes Topology Aware Routing does this.

Runs this way at: Slack

0 · Attribution, before any of it

Polar Signals built kubezonnet, an eBPF tool that traces packets and exposes per-zone flow metrics, because the cloud bill reports cross-zone transfer as an undifferentiated line and does not say which service produced it. They found it was near half their spend.

Runs this way at: Polar Signals

The divergence that matters: what happens when the local zone cannot serve

Layer 3 looks like one decision and is actually two. The first is "prefer the local zone". Every implementation does that. The second is "and here is what happens when the local zone is short, slow or wrong", and the implementations answer it completely differently.

Envoy's answer is capacity arithmetic performed continuously. Zone-aware routing has explicit preconditions, and the first one in the documentation is that "Both originating and upstream cluster are not in panic mode", panic mode being the state where fewer than half the hosts are healthy and Envoy starts ignoring health status altogether. When the local zone has proportionally less upstream capacity than the caller has callers, Envoy computes the fraction it can serve locally and distributes the remainder by residual capacity. That is a spill path, and it is driven by health.

Kubernetes' answer is a set of preconditions evaluated by a controller, and then nothing. The documentation lists five conditions under which hints are simply not applied, including "If there are less endpoints than zones in a cluster, the controller will not assign any hints", and a genuinely careful one about partial state: when some endpoints lack hints, "Filtering endpoints for a Service in this state would be dangerous so the kube-proxy falls back to using all endpoints." These are real safeguards and they are well designed for the failure they anticipate, which is imbalance. They are not a response to a zone becoming unhealthy while its endpoints continue to pass readiness probes, because kube-proxy is filtering an endpoint list, not measuring load. Envoy's own maintainers named the same limitation for their mechanism in February 2026, in a proposal to fix it: "It has no visibility into actual backend load."

TAR prevents pods in the failing zone from ever being able to reach pods in the other zones to compensate. William Morgan, Buoyant, 5 June 2024, describing a demonstration in which the failing zone drops to a 0% success rate while healthy pods sit idle elsewhere in the cluster

The reconstruction that follows, and neither Kubernetes nor AWS states it this way, is that these two designs are answering different questions. AWS's zonal services are built so that the whole path is zonal and a zone can be taken out of rotation as a unit at the layer above. Kubernetes Topology Aware Routing makes one hop zonal inside a cluster whose ingress, scheduler and autoscaler remain regional. Apply AWS's arithmetic to the Kubernetes case and the benefit largely evaporates: (2/3)^N only improves to 2/3 if every hop is local, and one local hop in a chain of five moves the exponent by one.

The decisions that matter

Five forks, each with what was chosen, what was rejected in public, and the condition that flips the answer. The flip conditions are derived from the stated reasons: where the reason is a constraint, the decision turns when the constraint does.

Figure 3 · Should this hop be zone-local?

Every terminal node is an action, not a preference. The two exits on the left are the ones the cost framing tends to skip. Derived from the thresholds in KEP-2433 and the preconditions in Envoy's documentation.

Diagram source

Decision 1: make this hop zone-local, or leave it regional?

Chosen

Zone-local, by DoorDash (Envoy mesh, 2023), Grab (Kafka follower fetch, 2023) and Slack (full siloing, 2021 onward)
DoorDash's saving was large enough that their cloud provider asked whether they were having an incident

Rejected

Leaving the mesh regional and paying, which is what all three were doing before
Rejected because the crossing count grows with call depth: "As the number of hops increases in microservice call graphs, the likelihood of data being transmitted across AZs grows"

Flips when

Fewer than three endpoints per zone, or an uneven spread, in which case Kubernetes will refuse and Envoy will skew
Traffic originates mostly from one zone, which the Kubernetes documentation names as a case where it "does not work well"
Your cloud does not charge, in which case you are buying only the risk

Decision 2: for a replicated log, follower fetch, rack-aware assignment, or object storage?

Chosen

Grab chose follower fetch (KIP-392) and measured a 25% cut in cross-AZ traffic after three months
Grafana Labs chose object storage, redesigning Mimir around a Kafka-compatible log with no inter-AZ networking cost

Rejected

KIP-392 rejected consumer-side replica selection: "it is easier to reason about the system in a multi-tenant environment when replica selection logic could be controlled by the broker"
Jack Vanlightly's October 2025 read of the Kafka debate: KIP-1183 "looks dead in the water, and not a serious contender"

Flips when

Your latency budget is the hinge. Grab measured up to +500 ms at p99 from follower fetch; Confluent's object-storage tier moves latency from sub-100 ms to "up to a second or two"
Below roughly a second of tolerance, follower fetch is the only option; above it, object storage removes the crossing and the replication decision together

Decision 3: endpoint filtering in kube-proxy, or health-aware locality in an L7 proxy?

Chosen

Kubernetes shipped endpoint filtering with hardcoded safeguards: hints only when expected overload is under 20%, redistribution above 30%
DoorDash and Slack both chose the mesh instead, carrying locality in xDS

Rejected

Making the overload threshold configurable per service was raised in October 2023 and the issue ran until April 2024
Combining node-local traffic policy with topology-aware routing was attempted in PR #103924 and closed unmerged after five and a half months
At GA in 1.33 only the hints field graduated; the routing behaviour was deferred

Flips when

You need spill driven by health rather than by endpoint counts, which requires load feedback kube-proxy does not have
You are already running a mesh, in which case the marginal cost of doing it at L7 is a control-plane change, not a new dependency

Decision 4: pre-provision for zone loss, or scale into it?

Chosen

AWS pre-provisions: with three zones "we overprovision by 50 percent", leaving each zone at 66% of tested load
Uber went the other way in 2026, cutting steady-state provisioning from 2× to 1.3× and lifting utilisation from 20% to 30%

Rejected

AWS rejects reactive scaling explicitly: it "relies on reacting to impairments as they happen, rather than being prepared for those impairments before they happen. In other words, it lacks static stability"
Uber rejected its own uniform model after four years of failover data

Flips when

You have measured the frequency rather than assumed it. Uber found full-peak regional failovers averaged under 20 hours a year, 0.23% of the time
And you have machinery to reclaim the buffer safely: Uber had to harden over 4,000 unsafe dependencies before the oversubscription was survivable

Decision 5: optimise the bill, or wait for the bill to change?

Chosen

Microsoft removed the charge entirely on 21 May 2024, framing it as encouraging "highly available and fault-tolerant system architectures"

Rejected

AWS and Google Cloud have not followed as of August 2026, at roughly $20 and $10.24 per terabyte respectively

Flips when

The optimisation is worth building only if it survives a repricing. Mechanisms that also improve gray-failure survival, such as full siloing with a drain, survive it. Mechanisms that only cut bytes do not

Decision	Chosen	Rejected	Because	Evidence
Zone affinity scope	Whole call path, siloed	One hop	Survival odds stay at 2/3 only if every hop is local	Builders' Library, 2019
Spill mechanism	Health and capacity driven	Endpoint-count driven	Counts do not detect a slow zone	Envoy #43665, 2026
Kafka consumer locality	Follower fetch plus rack-aware assignment	Consumer-side selection	Broker-controlled selection is easier to reason about in multi-tenant clusters	KIP-392
Kafka replication cost	Object storage under debate	KIP-1183 as the vehicle	Three proposals competed; one was judged not a serious contender	Vanlightly, 2025
Kubernetes option naming	`PreferSameZone`	`PreferClose`	Changing its semantics "would inevitably have either too many false positives or too many false negatives"	KEP-3015
Failover capacity	Tiered, oversubscribed	Uniform 2×	Full-peak failover is 0.23% of the year	Uber, 2026

What broke in production

Four failure classes account for everything in this corpus. Two are about the zone failing, two are about the affinity mechanism failing, and the second pair is almost entirely undocumented outside issue trackers, which is itself the finding.

Class A · The zone degrades without failing

This is the class the whole pattern exists for, and the class that health checks are worst at. Microsoft Research named the mechanism in 2017: gray failure, whose defining property is differential observability, meaning "the system's failure detectors may not notice problems even when applications are afflicted by them". Their section 2.1 is titled "High redundancy hurts", and they report that gray failure "is behind most cloud incidents".

Postmortem

Slack, 30 June 2021: an intermittent link poisons every zone

AssumptionA failing zone will look failed, and the load balancer will take it out.

What happened"A network link that connects one availability zone with several other availability zones containing Slack servers experienced intermittent faults." It was removed from service at 12:33pm PDT, returned, and failed again at 5:22pm. Because "a single Slack API request from a user may fan out into hundreds of RPCs to service backends, each of which must complete to return a correct response", a fractional error rate on one link produced user-visible failures across all zones.

Blast radiusErrors in every availability zone, not just the impaired one, across two separate windows on the same day.

FixSiloed cells, where a service "only receive[s] traffic from within their AZ and only send[s] traffic upstream to servers in their AZ", plus a drain that removes as much traffic as possible from a zone within five minutes, shifted at 1% granularity so in-flight requests complete.

Design ruleSiloing without a drain is half the pattern. Siloing contains the blast radius; only the drain ends the incident, and the drain is the part that needs a time budget and a rehearsal.

SourceSlack Engineering, 22 August 2023

Paper

Azure, 2017: adding redundancy lowered availability

AssumptionMore redundant paths between zones means more tolerance of failure in any one of them.

What happenedA Clos network reroutes around a switch that crashes. A switch that drops packets randomly and silently does not trigger rerouting, so every additional redundant path is another path that can silently degrade. "We sometimes see cases where increasing redundancy actually lowers availability."

Blast radiusReported across production Azure incidents rather than as a single event; the authors state gray failure is behind most cloud incidents.

FixClose the gap between what the failure detector observes and what the application observes, rather than masking it with more redundancy.

Design ruleTest your zone-affinity spill path against 5% packet loss, not against a killed pod. A killed pod fails the readiness probe; a degraded zone does not.

SourceHuang et al., HotOS '17

Figure 4 · The failure path that zone affinity does not cover

The readiness probe and the request take different paths through the degraded zone, which is why the endpoint stays in the list. Ordering reconstructed from Slack's account and the differential-observability definition in Huang et al., 2017.

Diagram source

Class B · The zone fails cleanly, and the dependency was not spread

The unglamorous class, and the one still producing multi-hour outages. On the night of 7 May 2026 multiple cooling units failed in a single data centre hall in AWS's use1-az4. AWS's statement, quoted by Network World, is that "servers automatically shut down when the temperatures exceeded the operating thresholds in order to protect the hardware", and that "EC2 instances and EBS volumes hosted on impacted hardware are affected by the loss of power during the thermal event".

Postmortem

AWS use1-az4, 7 to 10 May 2026: a thermal event, and a two-day control-plane outage downstream

AssumptionA zone impairment is a data-plane event that lasts hours.

What happenedCooling failed; servers shut themselves down; EC2 and EBS in that zone lost power. Cooling capacity was restored by 1:50pm PDT on 8 May, roughly 20 hours later. For Broadcom's VMware Cloud on AWS customers in that zone, the impact was not lost instances but lost control plane: "No new host provisioning", "No new SDDC deployments", and "Failure of auto-remediation for host failures".

Blast radiusBroadcom's incident began 02:02 UTC on 8 May and was not resolved until 00:49 UTC on 10 May, about 47 hours, well past AWS's own recovery.

FixAWS "shifted traffic away from the impacted zone for most services"; the downstream fix is the one AWS wrote in 2019, which is not to require the control plane in order to survive.

Design ruleRecovery that depends on provisioning is recovery that depends on the control plane, and the control plane is exactly what a zone event takes away. Pre-provision, or accept a recovery time set by someone else's queue.

SourceBroadcom status notice, Network World, 11 May 2026

Postmortem

AWS us-east-1, 19 to 20 October 2025: the outage multi-AZ does not help with

AssumptionSpreading across zones is the main defence against cloud failure.

What happened"A latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service's regional endpoint." Two DNS enactors ran concurrently, an older plan was applied late, the cleanup deleted it, and the regional endpoint lost all its addresses.

Blast radiusDynamoDB errors from 11:48pm PDT on 19 October to 2:40am on 20 October; EC2 launches failing until 1:50pm; NLB connection errors from 5:30am to 2:09pm; dependent services impaired through 2:20pm. Region-wide throughout.

FixDynamoDB DNS automation disabled worldwide pending a fix, velocity controls added to NLB health-check failover, EC2 propagation throttling improved.

Design ruleSize the spend on zone spreading against the frequency of zonal events specifically, not against cloud outages generally. Most of the headline events in this corpus were regional, and zone spreading bought nothing against them.

SourceAWS post-event summary

Class C · The optimisation silently turns itself off

This class has no postmortems because it does not cause outages. It causes a bill that goes back up, and nobody is paged for that. The mechanism is the safeguard working exactly as designed at a moment nobody is watching.

Source

Kubernetes #122437: scaling from 9 replicas to 11 removes every hint

AssumptionOnce topology-aware routing is on for a service, it stays on.

What happenedNine replicas spread [3, 3, 3] across three zones get hints. Scaling to eleven produces [4, 4, 3], the controller computes a negative remaining-minimum-endpoints value, and removes the hints with "Unable to allocate minimum required endpoints to each zone without exceeding overload threshold (11 endpoints, 3 zones)". A [4, 4, 3] spread is perfectly capable of serving same-zone traffic. Reported on v1.27.6 on AWS in December 2023.

Blast radiusCost, not availability, and invisible until the invoice. There is no default alert on hints being withdrawn.

FixIssue closed. The related request to make the threshold configurable per service ran from October 2023 to April 2024.

Design ruleAlert on the optimisation, not on its outcome. If a cost control can disable itself, the metric you need is "is it currently on", emitted per service, and it does not exist by default.

Sourcekubernetes/kubernetes #122437, #121516

Source

Kafka KAFKA-19387: the newer consumer protocol loses rack awareness

AssumptionUpgrading the client protocol preserves the cost optimisation configured under the old one.

What happenedRack-aware balancing in the default assignors has not yet landed for the newer consumer group protocol. The ticket to add it to the Range and Uniform assignors was opened on 7 June 2025 and is still In Progress with no fix version as of August 2026.

Blast radiusUnquantified publicly. The failure mode is consumers being assigned partitions whose leaders and followers are in other zones, restoring the traffic that client.rack was configured to avoid.

FixIn progress upstream; the workaround is writing your own assignor.

Design ruleA cost optimisation that lives in a client protocol is a cost optimisation with an upgrade path that can revoke it. Check it after every client or broker version bump, with a metric, not a config review.

SourceKAFKA-19387

Class D · The affinity mechanism skews the load it was meant to keep local

Source

Envoy #28419: one zone receiving five times the traffic of an equivalent zone

AssumptionZone-aware routing is either on or off, and distributes evenly among equivalent zones.

What happenedThree separate defects, reported 16 July 2023. Where local and upstream zone sets do not align, "zone-aware routing is enabled for some envoys in the local cluster, and disabled for others". Where the locality vector drops empty localities, residual weights are miscomputed such that "zone D will receive 5x as much traffic from the two local envoys in zone A". And an off-by-one in the residual-capacity threshold sends a sliver of weight to the wrong locality.

Blast radiusNot published as an incident. The reporter's framing is that "many backend services are scaled differently based on expected load", so the misalignment is the normal case rather than the edge case.

FixClosed, fixed via PR #28970.

Design ruleVerify the split empirically per zone after enabling affinity, and again after any change to the replica counts of either side. The configuration is symmetric; the behaviour is not.

Sourceenvoyproxy/envoy #28419

Eng blog

Grab, March 2023: follower fetching produced visible consumer skew

AssumptionReading from the nearest replica moves traffic without moving load.

What happenedAfter the rollout, Grab saw "visible load skewage" in consumer distribution from 1 March 2023, because the load a broker serves now depends on where the consumers happen to be, not on partition assignment. They also lost graceful broker isolation during maintenance: a demoted broker still serves followers, so consumers disconnect.

Blast radiusRequired provisioning additional brokers in the overloaded zones, and up to 500 ms of added end-to-end p99 latency.

FixExtra capacity in the skewed zones; the latency was accepted as the price, and the feature was kept off latency-sensitive use cases.

Design ruleZone affinity moves your load distribution from a variable you control, partition assignment, to one you do not, client placement. Budget capacity for the worst zone, not the average one.

SourceGrab Engineering, 7 July 2023

Where the record runs out

No public postmortem in this corpus attributes an outage to zone-affinity routing itself. Every class C and class D entry above comes from an issue tracker, a vendor demonstration or a project's own documented safeguards, not from an incident write-up with a timeline. There are two readings. Either the safeguards work and the failure genuinely does not occur, or it occurs and gets attributed to whatever it looked like at the top of the stack. Both readings point to the same action: measure the spill path yourself in a drill, because the public record cannot tell you it works.

Numbers you can plan against

Prices, shares, latency penalties and capacity ratios, each with the organisation it came from and the date it was true. Vendor claims are marked as claims.

Metric	Value	At	Context	As of	Source
Cross-zone transfer, AWS	$0.01/GB each way	AWS	$0.02 round trip; same-zone private-IP traffic is free	2023-04	Confluent
Cross-zone transfer, published as $/TB	$20 AWS, $10.24 GCP	Buoyant	Quoted in the reliability critique	2024-06	Buoyant
Cross-zone transfer, Azure	$0.00	Microsoft	Since 21 May 2024, private and public IPs; previously $0.01/GB	2026-08	Azure pricing
Networking share of Kafka infra cost	>50%	Confluent	Multi-AZ cluster; vendor's own observation across its fleet	2023-04	Confluent
Cross-zone share of total cloud spend	~50%	Polar Signals	Discovered mid-2024, before any optimisation	2025-01	eBPF Foundation
Cross-zone cost reduction achieved	50%	Polar Signals	After building per-flow attribution with eBPF	2025-01	eBPF Foundation
Cross-AZ traffic reduction, Kafka	25%	Grab	Three months after enabling follower fetch on Kafka 3.1	2023-07	Grab
Latency price of follower fetch	up to +500 ms p99	Grab	End to end; ruled the feature out for latency-sensitive topics	2023-07	Grab
Latency price of object-storage Kafka	sub-100 ms to 1–2 s	Confluent	Vendor claim for Freight clusters, alongside "90% lower infrastructure costs"	2025-02	Confluent
Static-stability overprovisioning	+50%	AWS	Three zones; each zone runs at 66% of tested load	2019	Builders' Library
Odds of avoiding an impaired zone	(2/3)^N vs 2/3	AWS	N regional hops versus N zonal hops in one call path	2019	Builders' Library
Time spent in full-peak failover	<20 h/yr, 0.23%	Uber	Four years of failover data across two regions	2026-03	Uber
Failover provisioning after redesign	2× to 1.3×	Uber	Utilisation 20% to 30%; about 1M of 4M CPU cores released	2026-03	Uber
Kubernetes hint thresholds	20% add, 30% redistribute	Kubernetes	Hardcoded; the 10-point gap exists to stop flapping	2021–2026	KEP-2433
Minimum endpoints for hints	≥ number of zones	Kubernetes	Below this, no hints at all; three pods across three zones can still be refused	2026-08	Kubernetes docs
Envoy panic threshold	50% healthy	Envoy	Below it, zone-aware routing is skipped entirely	2026-08	Envoy docs
Zone drain budget	5 min, 1% granularity	Slack	Target for removing traffic from an impaired AZ	2023-08	Slack
Zonal impairment duration	~20 h AWS, ~47 h downstream	AWS, Broadcom	use1-az4 thermal event, 7 to 10 May 2026	2026-05	Broadcom

A cost model with two variables

The model that matters has two inputs and one multiplier. The inputs are internal request bytes per second and replication factor. The multiplier is the fraction of hops that cross a zone, which in an evenly spread three-zone deployment starts at two thirds and compounds with call depth.

Derived, my arithmetic, shown so you can check it. One gigabyte per second sustained is 31.5 petabytes a year. At AWS's $0.01 per GB charged in one direction that is $315,360 a year; charged in both directions it is $630,720. Buoyant's published figure for the same rate is $410,000, which sits between the two and whose working is not shown. Use the unit price and your own measured byte count; treat any annualised headline figure, including that one, as an order of magnitude. The model excludes NAT gateway processing charges, load balancer capacity units and the cost of the extra replicas needed to keep zones balanced, all of which are real and none of which anyone in this corpus has published.

Read these carefully

The 90% infrastructure saving for Confluent Freight and the "zero inter-AZ cost" framing of object-storage streaming are vendor claims with no independent measurement in the public record. Grafana Labs corroborates the direction by adopting one of them but publishes no cost delta. The Confluent "over 50% of your Kafka infrastructure costs" figure is also the vendor's own, though it is arithmetically consistent with the traffic decomposition they publish alongside it. Every price in this table is a list price and none of them account for committed-spend discounts, which at large volumes is where the real number lives.

Figure 5 · Seven years of mechanisms aimed at one line item

The mechanisms move steadily up the stack, from a Kafka broker setting in 2019 to redesigning storage in 2025, while one cloud removed the incentive altogether in the middle of the sequence. Dates from the KIPs, KEPs and announcements cited in section 6.

Diagram source

The evidence wall

Thirty-four artefacts across twenty-two hosts, graded. The full ledger, with one row per claim and the copied quote behind each, ships beside this page as sources.md.

Postmortem Slack2023-08

Slack's Migration to a Cellular Architecture

The fullest public account of a zone gray failure and the architecture built in response. Names the mechanism (an intermittently faulting inter-AZ link), the amplifier (hundreds of RPCs per user request, all of which must succeed) and the fix (siloing plus a five-minute drain at 1% granularity).

Carry forwardSiloing contains; only the drain ends the incident. Give the drain a time budget and rehearse it.

slack.engineering/slacks-migration-to-a-cellular-architecture/

Postmortem AWS2025-10

Summary of the DynamoDB Service Disruption in US-EAST-1

A latent race condition between two DNS enactors left the regional DynamoDB endpoint with an empty record. Fourteen hours of cascading impact across EC2 launches, NLB and every dependent service, all of it regional in scope.

Carry forwardThe headline cloud outages in this corpus were regional. Zone spreading is insurance against a narrower event than most budgets assume.

aws.amazon.com/message/101925

Postmortem Broadcom2026-05

Availability Zone (use1-az4) Impacted

The downstream view of AWS's May 2026 thermal event. What broke for VMware Cloud on AWS was not running instances but the ability to provision: no new hosts, no new SDDCs, and auto-remediation for host failures stopped working. Open 02:02 UTC 8 May to 00:49 UTC 10 May.

Carry forwardA zone event removes the control plane you were counting on to recover from it. This is the concrete case for pre-provisioning.

status.broadcom.com/notices/xxpqglmvmv4dab7q-availability-zone-use1-az4-impacted

Postmortem Network World2026-05

AWS hit by US-East-1 outage after data center thermal event

Press reporting, quoting AWS's Health Dashboard verbatim on the 7 May 2026 cooling failure in use1-az4. Useful as the dated record of AWS's own words, since no formal post-event summary was published for this one at the time of checking.

Carry forwardSingle-zone physical failure is still a live event class in 2026, and it lasted roughly 20 hours.

networkworld.com/article/4168878/

Source Envoy2023-07

Issue #28419: zone-aware LB mishandles mismatched localities

Three defects in one report: the feature silently on for some proxies and off for others, a residual-weight miscalculation sending 5× traffic to one of two equivalent zones, and an off-by-one in threshold sampling. Closed via PR #28970.

Carry forwardVerify the per-zone split empirically after enabling affinity and after any replica-count change on either side.

github.com/envoyproxy/envoy/issues/28419

Source Envoy2026-02

Issue #43665: ORCA-driven locality routing proposal

The project stating its own limitation plainly: zone-aware routing "has no visibility into actual backend load", and "least-request helps within a zone but can't redistribute across zones". The proposal is to feed per-zone utilisation back into the split.

Carry forwardEndpoint-count-based affinity cannot detect a slow zone. If you need that, you need load reporting, and as of 2026 it is still being added.

github.com/envoyproxy/envoy/issues/43665

Source Kubernetes2023-12

Issue #122437: hints removed when scaling 9 to 11 replicas

A [3,3,3] spread gets hints, a [4,4,3] spread loses them, because the controller computes a negative remaining-minimum. Reported on v1.27.6 on AWS. The optimisation disables itself during a routine scale-up with no signal to the operator.

Carry forwardEmit and alert on whether hints are currently applied, per service. Nothing does this by default.

github.com/kubernetes/kubernetes/issues/122437

Source Kubernetes2023-10

Issue #121516: OverloadThreshold should be configurable per Service

Three Loki pods across three availability zones cannot get hints under the hardcoded 0.2 threshold, even though one pod could carry the whole load. Raised October 2023, closed as completed April 2024.

Carry forwardThe eligibility rule is about proportional capacity, not about whether same-zone routing would work. Small services are excluded by construction.

github.com/kubernetes/kubernetes/issues/121516

Source Kubernetes2021-07

PR #103924: traffic policy and topology-aware routing should not be exclusive

The rejected approach. An attempt to let node-local traffic policy and zone-aware routing coexist on the same Service, opened July 2021, closed unmerged in January 2022. The two features remain mutually exclusive in the documentation today.

Carry forwardIf you need both node-locality and zone-locality, Kubernetes will not give them to you together. That constraint is five years old and still binding.

github.com/kubernetes/kubernetes/pull/103924

Source Apache Kafka2025-06

KAFKA-19387: rack awareness in Range and Uniform assignors

Rack-aware balancing has not yet landed for the newer consumer group protocol's default assignors. Opened 7 June 2025, still In Progress with no fix version. Four sub-tasks covering homogeneous and heterogeneous assignment.

Carry forwardA protocol upgrade can revoke a cost optimisation configured under the previous one. Re-verify after every version bump, with a metric.

issues.apache.org/jira/browse/KAFKA-19387

ADR Apache Kafka2019

KIP-392: Allow consumers to fetch from closest replica

The decision record that started the practice, and it is explicit about motive: consumers could only fetch from the leader, "so there is no easy way to leverage locality in order to reduce expensive cross-dc traffic", in a deployment where "each availability zone is treated as a datacenter". Documents the correctness work required: high-watermark propagation, out-of-range offset handling, and a new error code.

Carry forwardReading from a follower is not free correctness. It buys cost with latency and with "spurious out of range errors" on out-of-sync replicas.

cwiki.apache.org/confluence/display/KAFKA/KIP-392

ADR Apache Kafka2022

KIP-881: Rack-aware Partition Assignment for Kafka Consumers

The second route to the same outcome: rather than redirecting the fetch, align the assignment so the consumer's partitions already live in its zone. Propagates client.rack to the assignor.

Carry forwardTwo mechanisms, one goal, different failure modes. Assignment alignment avoids the follower-fetch latency penalty but constrains the assignor.

cwiki.apache.org/confluence/spaces/KAFKA/pages/231118120/

ADR Apache Kafka2025-05

KIP-1183: Unified Shared Storage

One of three competing proposals to remove cross-zone replication from Kafka altogether by putting the log on object storage. Notes that "the community has made significant efforts to reduce cross-AZ traffic" already, and that with shared storage "data replication is unnecessary". Still Under Discussion.

Carry forwardThe line item is now large enough to be reopening the storage design of a mature system. Watch which proposal wins before committing.

cwiki.apache.org/confluence/display/KAFKA/KIP-1183

ADR Kubernetes2021-02

KEP-2433: Topology Aware Hints

Where the hardcoded thresholds come from, and why: a 30% maximum acceptable overload, hints only added below 20% expected overload, the 10-point gap existing to stop flapping. The risks section anticipates the single-zone-traffic case exactly. At GA in 1.33 only the hints field graduated.

Carry forwardThe KEP predicted the overload risk and the mitigation it offers is "deploy separately per zone", which is a different architecture, not a setting.

github.com/kubernetes/enhancements/.../2433-topology-aware-hints

ADR Kubernetes2025

KEP-3015: PreferSameZone and PreferSameNode

A project renaming its own feature because the first name overpromised. Extending PreferClose to cover node locality "would inevitably have either too many false positives or too many false negatives", so the KEP deprecates it and replaces it with a name that says what it does.

Carry forward"Close" is not a property a routing layer can evaluate. Zone and node are. Prefer configuration that names the topology level explicitly.

github.com/kubernetes/enhancements/.../3015-prefer-same-node

Paper Microsoft Research2017-05

Gray Failure: The Achilles' Heel of Cloud-Scale Systems, HotOS '17

Defines differential observability, the property that makes a degraded zone invisible to the thing responsible for removing it. Section 2.1, "High redundancy hurts", is the direct argument against assuming that spreading across zones is by itself protective.

Carry forwardDesign the zone-health signal from the caller's perspective, not the endpoint's. A readiness probe and a real request take different paths.

microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf

Paper Uber, UBC2026-03

Uber's Failover Architecture, arXiv:2603.07345

The same trade one level up: standing redundancy against a rare event. Four years of data showed full-peak failovers averaging under 20 hours a year. Uber moved from 2× to 1.3× provisioning, lifted utilisation from 20% to 30%, and released about a million CPU cores from a four-million-core baseline, after hardening over 4,000 unsafe dependencies.

Carry forwardMeasure how often the event you are provisioning against actually happens, then decide. Uber found the answer was 0.23% of the year.

arxiv.org/pdf/2603.07345

Eng blog Buoyant2024-06

The trouble with Topology Aware Routing

The clearest statement of the counter-case, with a demonstration: with TAR enabled and one zone's backends failing, that zone drops to a 0% success rate while healthy pods sit idle elsewhere. Also carries the published unit prices, $20/TB on AWS and $10.24/TB on Google Cloud.

Carry forwardRun this experiment on your own cluster before enabling zone-local routing. It takes an afternoon and it is the only thing that tells you your spill path exists.

buoyant.io/blog/the-trouble-with-topology-aware-routing...

Eng blog Grab2023-07

Zero traffic cost for Kafka consumers

The most complete numbers in the corpus for what follower fetching actually costs and returns: a 25% cut in cross-AZ traffic after three months, up to 500 ms added at p99, visible consumer skew from March 2023, and the loss of graceful broker isolation during maintenance.

Carry forwardBudget for extra brokers in the zone your consumers happen to cluster in. Affinity moves load distribution to a variable you do not control.

engineering.grab.com/zero-traffic-cost

Eng blog Grafana Labs2025-02

The next generation of Grafana Mimir

An architecture changed by a pricing line. Multi-AZ was already possible in the old design, but "the cross-availability zone data transfer charges it incurred made this cost-prohibitive to do on a large scale", so the write path was rebuilt on a Kafka-compatible log over object storage. Load tested at 1.5 GiB/s compressed writes with 4× fan-out.

Carry forwardWhen the crossing cannot be routed around, the remaining move is to relocate durability into a store that is already regional.

grafana.com/blog/the-next-generation-of-grafana-mimir...

Eng blog Jack Vanlightly2025-10

A Fork in the Road: Deciding Kafka's Diskless Future

An independent read of three simultaneous KIPs aimed at the same cost. Quotes Luke Chen on "three KIPs simultaneously addressing the same challenge of high replication costs when running Kafka across multiple cloud availability zones", and warns against the narrow framing: "it's a mistake to think of S3 simply as a cheaper disk or a networking cheat".

Carry forwardThe cost framing understates what changes. Object-storage logs alter elasticity and operations too, and those may matter more than the bill.

jack-vanlightly.com/blog/2025/10/22/a-fork-in-the-road...

Eng blog InfoQ2024-06

Microsoft Drops Data Transfer Fees between Availability Zones

The record of the pricing change and Microsoft's stated reason: removing the charge "encourages the use of highly available and fault-tolerant system architectures without additional fees". Includes the comparison to AWS at $0.01/GB per direction.

Carry forwardAny architecture whose only justification is this charge is an architecture with a repricing risk. Check whether your mechanism also buys reliability.

infoq.com/news/2024/06/azure-az-transfer-fees/

Case study DoorDash2024-01

DoorDash uses a service mesh to manage data transfer and cloud spend

Zone awareness carried in the mesh control plane, with each node's availability zone placed into endpoint discovery responses. The outcome, in the engineers' own words: the saving "caused our cloud provider to reach out to us asking whether we were experiencing a production-related incident".

Carry forwardDoing this at L7 keeps the health-driven spill path that endpoint filtering gives up. If you already run a mesh, that is the cheaper place to do it.

infoq.com/news/2024/01/doordash-service-mesh

Case study Polar Signals2025-01

Using eBPF to monitor cross-zone traffic, cutting those costs by 50%

Attribution before optimisation. Cross-zone traffic was nearly half their cloud bill in mid-2024, and the cloud invoice does not say which service produced it, so they built kubezonnet to trace packets and expose per-flow Prometheus metrics.

Carry forwardYou cannot prioritise this work from the invoice. Per-service cross-zone byte attribution is the prerequisite, and it is a build, not a setting.

ebpf.foundation/case-study-polar-signals-uses-ebpf...

Talk SignalFx, Kafka Summit SF2019

Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka

Uday Sagar Siramshetty's session, from the same year as KIP-392. The framing has not changed in seven years: clouds allow free transfer inside a zone and bill for movement between them, so the architecture is bent to produce and consume within one zone. Also covers compression during replication.

Carry forwardThe 50% figure predates every mechanism now sold as the solution. Compression on the replication path is the option most teams still forget.

confluent.io/kafka-summit-san-francisco-2019/achieving-a-50-reduction...

Talk KubeFM2025-01

Topology-aware routing: balancing cost savings and reliability

William Morgan of Buoyant, in conversation, seven months after the written critique. Useful for hearing the trade-off argued rather than asserted, and for the point that the Kubernetes safeguards protect against imbalance rather than against a degraded zone.

Carry forwardThe disagreement is still live in the practitioner channel a year after the feature shipped broadly. Do not treat either side as settled.

kube.fm/topology-aware-routing-william

Vendor AWS2019

Static stability using Availability Zones, Amazon Builders' Library

Becky Weiss and Mike Furr. The origin of the reliability argument for zone-local traffic, with the arithmetic: (2/3)^N for N regional hops against a constant 2/3 for zonal ones. Also the capacity price, "when we use three Availability Zones, we overprovision by 50 percent", and the explicit rejection of reactive scaling.

Carry forwardThe reliability case only holds for a fully zonal path. One zone-local hop in a regional chain moves the exponent by one and little else.

d1.awsstatic.com/builderslibrary/pdfs/static-stability-using-availability-zones.pdf

Vendor Kubernetes2026-08

Topology Aware Routing (documentation)

The five conditions under which hints are not applied, including fewer endpoints than zones, missing zone labels, and the careful partial-state rule: filtering in that state "would be dangerous so the kube-proxy falls back to using all endpoints". Also states the feature "does not work well" when traffic originates mostly from one zone.

Carry forwardRead this page as a list of the ways the optimisation turns itself off, and instrument each one.

kubernetes.io/docs/concepts/services-networking/topology-aware-routing/

Vendor Envoy2026-08

Zone aware routing (documentation)

The preconditions, of which the first is that neither cluster is in panic mode, panic being the state below 50% healthy hosts where Envoy ignores health entirely. Documents the residual-capacity split when the local zone cannot serve, and that zone-aware routing applies only at priority zero.

Carry forwardEnvoy's affinity is conditional on health by construction, which is the property endpoint-list filtering lacks. That is the substantive difference between the two approaches.

envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/load_balancing/zone_aware

Vendor Confluent2023-04

Uncovering Kafka's Hidden Infrastructure Costs

Addison Huddy's decomposition of where cross-zone bytes come from in a replicated log: producers reaching a remote leader about two thirds of the time, replication at factor three adding twice the ingress, consumers reading remotely about two thirds of the time. Claims networking is over half of Kafka infrastructure cost.

Carry forwardThe decomposition is reusable arithmetic even if you discount the headline share. Apply it to your own replication factor and fan-out.

confluent.io/blog/understanding-and-optimizing-your-kafka-costs-part-1-infrastructure/

Vendor Confluent2025-02

Confluent Cloud Freight Clusters in GA

The commercial form of the trade, and useful mainly for the latency number the vendor is willing to publish: from "sub-100ms" in standard clusters to "up to a second or two" in exchange for removing cross-zone traffic cost. The 90% infrastructure saving is a claim with no independent measurement.

Carry forwardTreat one to two seconds as the honest price of removing the crossing from a log entirely. If your consumers cannot absorb that, this route is closed.

confluent.io/blog/freight-clusters-are-generally-available/

Vendor Microsoft Azure2026-08

Bandwidth pricing

The current price, confirmed at the source: transfer within and between availability zones in the same region is free, as is transfer between Azure services in the same region. Cross-region charges remain.

Carry forwardIf you are on Azure, most of the engineering in this guide is negative value: you would take on the failure modes and get no saving.

azure.microsoft.com/en-us/pricing/details/bandwidth/

Vendor AWS2023-05

Exploring the effect of Topology Aware Hints in Amazon EKS

AWS's own walkthrough, and worth noting for what it does not contain: no before and after percentages, only X-Ray traces showing that each UI pod stops talking to catalog pods in other zones. Does warn that horizontal pod autoscaling can disrupt zone balance on scale-in.

Carry forwardEven the cloud vendor's demonstration is qualitative. Nobody has published a measured cross-zone byte reduction for this feature in isolation.

aws.amazon.com/blogs/containers/exploring-the-effect-of-topology-aware-hints...

Vendor WarpStream2025-02

How WarpStream powers Grafana Labs' redesigned architecture

The supplier's account of the Grafana Mimir redesign, reproduced with permission. Adds the load-test figures the customer post carries too: 1.5 GiB/s compressed writes through a single cluster, consumed with 4× fan-out for 7.5 GiB/s aggregate. The "no inter-AZ networking costs" framing is the vendor's.

Carry forwardUseful as corroboration that a named production system took this route at a stated throughput, not as evidence of the saving, which neither side publishes.

warpstream.com/blog/how-warpstream-powers-grafana-labs-redesigned-architecture

Build a miniature, then productionise it

Seven rungs. The first three are an afternoon each on a scratch cluster. The line from toy to production-shaped is crossed at rung four, where you stop measuring the saving and start measuring the recovery.

Measure the bytes before changing anything

Stand up a three-zone cluster, run a load generator through a two-hop service chain, and attribute cross-zone bytes per service pair. VPC flow logs will do it; an eBPF agent does it with less noise. Polar Signals built kubezonnet for exactly this because the invoice reports the total and not the cause.

Done when: you can state, per service, what fraction of its egress crosses a zone boundary. Teaches: that the default is roughly two thirds, and that nothing in the cloud console will tell you which workload is responsible.

Turn on zone-local routing and watch it disable itself

Set trafficDistribution: PreferSameZone on the service, confirm the hints appear on the EndpointSlices, then scale the deployment until they vanish. Reproduce issue #122437's [3,3,3] to [4,4,3] transition, and find the controller log line that explains it.

Done when: you have the log line and the exact replica count at which hints disappear for your zone layout. Teaches: the optimisation is conditional, the condition is arithmetic on endpoint counts, and its removal is silent.

Kill a zone's backends with affinity on

Scale the backend deployment in one zone to zero, or blackhole it, while clients in that zone keep sending. Record success rate by client zone, not in aggregate. This is Buoyant's experiment and it takes under an hour once the harness exists.

Done when: you have a per-zone success-rate chart across the event and can say whether traffic spilled or dropped. Teaches: whether your configuration has a spill path at all, which is not answerable from the configuration itself.

Build the drain, and put a stopwatch on it

Add a control that moves all traffic out of one zone and back. Slack's bar is five minutes at 1% granularity so in-flight requests complete. Wire it to something a human can run at 3am without reading a runbook, and record how long it actually takes on your system.

Done when: a single command drains a zone in under five minutes and restores it, twice in a row, with no error spike. Teaches: the fifth layer of the affinity stack, and the difference between containing an incident and ending one.

Run the gray failure, not the clean one

Inject 5% packet loss and 200ms of added latency on one zone's paths rather than killing anything. Readiness probes will keep passing. Watch whether any part of your stack notices, and whether the drain from rung four gets triggered by anything other than a human reading a graph.

Done when: a degraded, non-failing zone produces a distinct alert within your detection budget. Teaches: differential observability, concretely, on your own system rather than in a paper.

Do the same for the stateful tier, and price the latency

Enable client.rack and RackAwareReplicaSelector on a Kafka cluster, then measure two things: the change in cross-zone bytes, and the p99 delta. Grab measured 25% and up to 500ms respectively. Also chart consumer-to-broker distribution per zone, because that is where the skew shows up.

Done when: you have your own numbers for both, and a per-zone broker load chart. Teaches: that the stateful tier pays in latency and in capacity headroom, not just in configuration.

Write the business case, including what it costs to keep

Put the measured saving next to the extra replicas needed to keep zones balanced, the engineering time, the alerting you had to build in rungs two and five, and the repricing risk. Then apply Uber's question: how often does the event you are provisioning against actually happen to you, in hours per year?

Done when: you can defend or withdraw the change in a review with numbers on both sides. Teaches: that the saving is the easy half, and that a cost control which needs its own alerting is not free.

Keep hunting

The queries that actually produced the material above, grouped by what they surface. The vocabulary is the useful part: "cross-AZ tax", "follower fetching", "locality load balancing", "siloed", "static stability" and "gray failure" each open a different seam.

Production experience and numbers

"cross-AZ" data transfer cost "we reduced" production
"cross-zone" cost "we cut" OR "we reduced" engineering blog
"zero traffic cost" OR "cross-AZ tax" kafka OR kubernetes
site:engineering.grab.com OR site:careersatdoordash.com availability zone cost

The decision records

KIP-392 "fetch from closest replica" rejected alternatives
path:keps/sig-network topology aware hints overload threshold
"KIP-1150" OR "KIP-1176" OR "KIP-1183" diskless replication cost
"trafficDistribution" "PreferSameZone" KEP rationale deprecates

Where it goes wrong

repo:kubernetes/kubernetes is:issue "topology aware" overload threshold
repo:envoyproxy/envoy is:issue "zone aware routing" skew OR locality mismatch
repo:kubernetes/kubernetes is:pr is:closed is:unmerged topology aware
"topology aware routing" "we disabled" OR "rolled back" incident

Zone failure itself

"gray failure" availability zone drain siloed incident
use1-az4 OR eun1-az3 impacted status page incident
"static stability" availability zones overprovision independence
site:aws.amazon.com/message post-event summary region

Two chains produced most of the value here and both are reusable. The first was vocabulary chaining: the phrase "follower fetching" came out of a Kafka KIP, and searching that phrase rather than "kafka cost" found Grab, SignalFx and the Confluent decomposition within one round. The second was reading a project's own safeguards as a failure catalogue. The Kubernetes documentation page listing five conditions under which hints are not applied is, read the other way, a list of five ways your cost optimisation turns itself off, and each of those conditions has an issue thread behind it with a real user attached.

References

Cooper Bethea, Slack's Migration to a Cellular Architecture Slack Engineering, 22 August 2023. Checked 2026-08-30.
Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region AWS, October 2025. Checked 2026-08-30.
Availability Zone (use1-az4) Impacted Broadcom Service Status, 8 to 10 May 2026. Checked 2026-08-30.
Gyana Swain, AWS hit by US-East-1 outage after data center thermal event Network World, 11 May 2026. Checked 2026-08-30.
Zone-aware load balancing incorrectly handles mismatched local and upstream localities, issue #28419 envoyproxy/envoy, 16 July 2023. Checked 2026-08-30.
Proposal: ORCA-driven locality routing for zone-aware load balancing, issue #43665 envoyproxy/envoy, 26 February 2026. Checked 2026-08-30.
Topology Aware Routing: Unable to allocate minimum required endpoints, issue #122437 kubernetes/kubernetes, 21 December 2023. Checked 2026-08-30.
Topology Aware Routing's OverloadThreshold should be configurable per Service, issue #121516 kubernetes/kubernetes, 25 October 2023. Checked 2026-08-30.
Traffic Policy and Topology Aware Routing should not be exclusive, PR #103924, closed unmerged kubernetes/kubernetes, 26 July 2021 to 14 January 2022. Checked 2026-08-30.
KAFKA-19387: Support rack awareness in Range and Uniform assignors Apache Kafka JIRA, 7 June 2025. Checked 2026-08-30.
KIP-392: Allow consumers to fetch from closest replica Apache Software Foundation, 2019. Checked 2026-08-30.
KIP-881: Rack-aware Partition Assignment for Kafka Consumers Apache Software Foundation, 2022. Checked 2026-08-30.
KIP-1183: Unified Shared Storage Apache Software Foundation, updated 16 May 2025. Checked 2026-08-30.
KEP-2433: Topology Aware Hints kubernetes/enhancements, alpha 1.21 through GA 1.33. Checked 2026-08-30.
KEP-3015: PreferSameZone and PreferSameNode Traffic Distribution kubernetes/enhancements, 2025. Checked 2026-08-30.
Peng Huang, Chuanxiong Guo, Lidong Zhou, Jacob R. Lorch, Yingnong Dang, Murali Chintalapati, Randolph Yao, Gray Failure: The Achilles' Heel of Cloud-Scale Systems HotOS '17, Whistler BC, May 2017. Checked 2026-08-30.
Mayank Bansal et al., Uber's Failover Architecture: Reconciling Reliability and Efficiency in Hyperscale Microservice Infrastructure arXiv:2603.07345, 7 March 2026. Checked 2026-08-30.
William Morgan, The trouble with Topology Aware Routing Buoyant, 5 June 2024. Checked 2026-08-30.
Fabrice Harbulot and Quang Minh Tran, Zero traffic cost for Kafka consumers Grab Engineering, 7 July 2023. Checked 2026-08-30.
Zhehao Zhou, The next generation of Grafana Mimir Grafana Labs, 21 February 2025. Checked 2026-08-30.
Jack Vanlightly, A Fork in the Road: Deciding Kafka's Diskless Future 22 October 2025. Checked 2026-08-30.
Eran Stiller, DoorDash Uses Service Mesh to Manage Data Transfer and Reduce Cloud Spend InfoQ, 23 January 2024. Checked 2026-08-30.
Renato Losio, Microsoft Drops Data Transfer Fees between Availability Zones InfoQ, 8 June 2024. Checked 2026-08-30.
Case Study: Polar Signals Uses eBPF to Monitor Internal Cross-Zone Network Traffic on Kubernetes eBPF Foundation, updated 15 January 2025. Checked 2026-08-30.
Uday Sagar Siramshetty, Achieving a 50% Reduction in Cross-AZ Network Costs from Kafka Kafka Summit San Francisco, 2019. Checked 2026-08-30.
Topology-aware routing: balancing cost savings and reliability, with William Morgan KubeFM, 21 January 2025. Checked 2026-08-30.
Becky Weiss and Mike Furr, Static stability using Availability Zones Amazon Builders' Library, 2019. Checked 2026-08-30.
Topology Aware Routing Kubernetes documentation. Checked 2026-08-30.
Zone aware routing Envoy documentation. Checked 2026-08-30.
Addison Huddy, Uncovering Kafka's Hidden Infrastructure Costs Confluent, 20 April 2023. Checked 2026-08-30.
Confluent Cloud Freight Clusters in GA Confluent, 3 February 2025. Checked 2026-08-30.
Bandwidth pricing Microsoft Azure. Checked 2026-08-30.
Bingjiao Yu, Niall Thomson, Xiangyan Wang, Exploring the effect of Topology Aware Hints on network traffic in Amazon EKS AWS Containers Blog, 16 May 2023. Checked 2026-08-30.
How WarpStream Powers Grafana Labs' Redesigned Architecture WarpStream, 23 February 2025. Checked 2026-08-30.