Service Mesh Platform  ·  View 18 of 31  ·  5 · Runtime

A Canary That Fails

checkout v2.14 is started at 1%, its error ratio breaches, and the rollout reverts itself without a human touching a route.

Editable source SVG draw.io All views
Release engineer Argo Rollouts Kubernetes API istiod Caller proxies Thanos 1. start v2.14 at 1% 2. route weights 99 / 1 3. watch event 4. delta xDS: one route 5. ACK 6. applied canary share? error ratio? 7. 0.98% · 5xx 2.3× stable 8. analysis failed: abort 9. route weights 100 / 0 10. delta xDS: one route 11. aborted · 41 s after breach A Canary That Fails — Automatic Rollback in 41 Seconds The rollout checks the share the mesh reports, not the weight it wrote. Only Argo Rollouts may write this route; a human aborts the rollout, not the route. v 1.0 · owner Platform Networking Architecture · date 2026-09

Decisions

  • Analysis runs on a 30-second interval and aborts on two consecutive failures. The 41 seconds includes detection; the revert itself is one route change delivered as a delta.
  • The rollout reads the canary share from mesh metrics before judging error ratio. If the applied share is outside tolerance, the analysis is inconclusive and the rollout pauses instead of judging a canary that received no traffic.
  • A person aborts the Rollout object. Admission guards reject direct edits to a route owned by an active rollout, which is what stops a rollback racing a rollout.

Targets

  • Rollback complete ≤ 60 s after breach detection. Weight change applied fleet-wide p95 ≤ 10 s.

Risks

  • At 1% of a low-traffic service, an analysis window may hold too few requests to judge. Rollouts declare a minimum request count per step; below it the step waits rather than guesses.