Service Mesh Platform  ·  View 23 of 31  ·  6 · Operations

Configuration Rollout in Waves

A merged mesh change reaches one canary cluster first, then one cluster per site, then the rest, with a health gate between each wave.

Editable source SVG draw.io All views
Merge Intent change owner repo · 2 approvals Wave 0 Canary cluster synthetic + real Health gate 10 min Wave 1 One cluster per site ≈ 15% of proxies Health gate 15 min Wave 2 Remaining clusters site by site Health gate per site On breach Pin previous bundle one operation · ≤ 60 s breach A Mesh Configuration Change Rolls Out in Waves Data store Application we own Decision point Risk / gap failure / alternate Gates read mesh-attributed error ratio, NACK count and drift. Wave 1 and 2 gates halt and revert the same way; one edge is drawn. v 1.0 · owner Platform Networking Architecture · date 2026-09

Decisions

  • Waves are ordered by cluster, not by proxy. A cluster is the unit with its own control plane, so a bad change is contained by the same boundary that contains a bad istiod.
  • Gates read three signals: mesh-attributed error ratio, NACK count and proxies still on the previous sha. Application error rates are shown but do not gate, or every unrelated incident would halt every change.
  • On breach the wave halts and the affected clusters are pinned to the previous rendered bundle. Git is reverted afterwards, by the same operation, so Argo CD does not re-apply the bad change.

Exception

  • An emergency deny skips waves and applies to every cluster at once. It is the one change type where speed is worth more than staging, and it is audited as an exception.

Targets

  • Full rollout across 20 clusters in about 45 minutes. Halt and revert ≤ 60 s from breach.