Service Mesh Platform  ·  View 31 of 31  ·  7 · Assurance

Failure Modes

The ten failure classes the requirement assumes will happen, how each is noticed, what contains it, and what requests see meanwhile.

Editable source SVG draw.io All views
Detected by Contained by Requests see Back to normal istiod down xDS age per proxy Last-good config held Nothing Push resumes SPIRE or CA down Renewal errors · expiry 24 h SVIDs · 7 d CA Nothing for 12 h Emergency issuance Bad config push Wave gate · NACKs Wave 0 only Errors in one cluster Bundle pinned ≤ 60 s Trust-bundle skew Handshake cause code Additive bundle · 30 d TLS errors, attributed Bundle converges Proxy OOM Container restarts Pod unready · drained Retried on a peer Restart refetches Injection webhook down Admission errors Strict namespaces refuse Nothing · no new pods Webhook back Retry storm retry_overflow Budget 20% · limits Fast 503s Budget refills East-west gateway lost Remote ejections Local-only serving Less remote capacity Gateway returns Clock skew chrony offset over 60 s Node tainted Nothing Resync · untaint Telemetry sink slow Collector drop count Bounded buffers Nothing Gap in telemetry Failure Modes — Detection, Containment, What Requests See v 1.0 · owner Platform Networking Architecture · date 2026-09

Pattern

  • In seven of ten rows requests see nothing or a fast, attributed failure. The three that reach users (a bad push, lost remote capacity, a proxy crash) are each bounded to one cluster, one destination or one pod.
  • Every detection column is a signal that exists before the failure matters: xDS age before staleness hurts, expiry before certificates lapse, drop counts before telemetry gaps are noticed.

Declared per namespace

  • Injection webhook failure refuses pods in strict namespaces and admits them unmeshed in permissive ones. There is no global default, and an admission policy independent of the webhook refuses an unmeshed pod in a strict namespace.

Drills

  • Each row has a quarterly game-day on the staging mesh. The control-plane outage and the bad config push also run once a year on a production cluster in wave 0.
◀  Authorisation at the Destination All views
Solution Architecture v1.0 · Platform Networking Architecture · 2026-09 · 31 views · open source, fully on-premises31-failure-modes