After a mesh rollout, one service's p99 rises from 30 ms to 70 ms, but only for calls to services in other namespaces. Same-namespace calls are unchanged. CPU on both sides is flat and the proxy reports no errors. What is the most likely cause?
Show the full answer Hide the answer
The first three things to look at
- What is different about the affected calls. They cross a namespace boundary, which in most clusters correlates with crossing a team boundary and, importantly, with a different set of endpoints spread across zones.
- The shape of the increase. 40 ms added at p99 with a flat p50 says a subset of requests is taking a longer path, not that every request got slower. A uniform overhead would lift p50 too.
- Endpoint distribution per zone, from the proxy's own statistics. If the caller in zone A is sending a third of its traffic to each of three zones, the answer is in front of you.
The diagnosis
Before the mesh, the client called a cluster service name and got whatever the platform's load balancing did, which in many setups favoured local endpoints. The mesh replaced that with its own load balancing across the full endpoint set, and by default it balances for even distribution rather than for proximity. Cross-zone round trips inside a region commonly add on the order of one to a few milliseconds per hop, and a call chain of several hops that each cross a zone accumulates into exactly the tail seen here. Same-namespace services happened to be co-located, so their calls never left the zone.
There is a second bill attached: cross-zone traffic is charged in most clouds, so this shows up in the invoice as well as in the tail.
The fix
Enable locality-aware routing so the proxy prefers endpoints in its own zone and fails over to other zones only when local ones are unhealthy or insufficient. Keep the failover, because pinning entirely to one zone converts a latency win into a single-zone dependency, which is the opposite of what the platform is for.
Why the other options fail
- Proxy CPU throttling is a genuine and very common mesh problem, and it is ruled out here by the evidence: throttling is indifferent to the destination namespace, so it would raise latency for all calls. Check it anyway, since container limits on sidecars are frequently set too low, but it does not explain this shape.
- Handshakes per request is not how mTLS works. The handshake is per connection and proxies pool connections. If connection reuse were broken you would see the cost on new connections only and a matching rise in connection counts, which is a different and visible signal.
- DNS retried per call misreads where names are resolved. The proxy routes on the destination service from its own configuration and does not perform a DNS lookup per request. DNS problems in a mesh show up as convergence failures and outright errors, not as a clean 40 ms tail.
The alert that would have caught it earlier
Cross-zone request ratio per service, alongside p99 by destination zone. Both are available from proxy statistics on day one of a mesh rollout and neither is on a default dashboard. The rollout checklist should include a before-and-after comparison of the zone distribution, because this regression is introduced by the mesh at the moment it takes over load balancing.
When this is the wrong hypothesis
In a single-zone cluster none of this applies, and the same symptom would point at proxy resource limits instead. The zone hypothesis is only worth chasing when the endpoint set actually spans zones, which is the first thing to check rather than the last.