Service Mesh Platform  ·  View 05 of 31  ·  2 · People and journeys

Journey — Canary a Release at 1%

A release engineer ships checkout v2.14. The anxious moment is the first 1% of real traffic, and the question is whether the mesh did what it was told.

Editable source SVG draw.io All views
Release engineer checkout v2.14 Goal — Find a bad build on 1% of traffic Trigger — Merged change to the pricing path Done when — Promoted, or reverted inside 60 s 1 · Declare 2 · Mirror 3 · Cohort 4 · 1% live ◆ moment of truth 5 · Widen 6 · Promote What they do Opens the rollout Mirrors reads to v2.14 Routes staff by header Starts general 1% Steps 5 → 25 → 50% Promotes to 100% System behind it Argo Rollouts Mirror route Header match Weighted route Analysis on Thanos Stable subset retired How it feels Confident Unsure Anxious Where it hurts Mirror writes twice? Did 1% really apply? p99 up 40 ms at 25% What the platform gives One owner of weights Only GET mirrored Named cohort first Applied share read back Auto abort ≤ 60 s Every step revertible Journey — A Release Engineer Canaries checkout v2.14 at 1% v 1.0 · owner Platform Networking Architecture · date 2026-09

The trough, and what answers it

  • "Did 1% really apply?" The rollout does not trust the weight it wrote. Its analysis reads the canary's share of requests from mesh telemetry, and it refuses to advance if the share is outside a tolerance around the target.
  • p99 rising at 25% is caught by analysis against the stable subset in the same window, not against yesterday's baseline, so traffic-mix changes do not trigger false aborts.

Decisions behind it

  • Argo Rollouts owns weights during a rollout. Admission guards deny writes to that route from any other identity, so a human cannot race the controller (ADR-15).
  • Mirroring is allowed only for methods declared idempotent. A mirrored POST to a payment service would create a second payment.

Targets

  • Weight change applied fleet-wide p95 ≤ 10 s. Automatic rollback complete ≤ 60 s after a breach is detected.
◀  Journey — Move a Service to Strict mTLS All views Journey — Contain a Compromised Workload  ▶
Solution Architecture v1.0 · Platform Networking Architecture · 2026-09 · 31 views · open source, fully on-premises05-journey-canary-a-release-at-one-percent