Edge Cache and CDN Platform  ·  View 19 of 29  ·  5 · Runtime

A Configuration Change, Halted and Reverted

A rule passes validation and still makes things worse at the canary. The ring controller notices, stops and reverts without a person deciding.

Editable source SVG draw.io All views
Property owner Config repo Compiler + validator Ring controller Canary PoP · WAW Thanos · ClickHouse 1. merge request · new rule /api/v2 2. CI: CUE schema · invariants 3. identity probe: two test users, same body? 4. pass · bundle 4be1 signed 5. start rollout 4be1 6. activate 4be1 7. applied in 9 s 8. 10 min: 5xx, hit ratio, stale ratio 9. hit ratio −6 points on /api/v2 10. regression > 2 points → halt 11. revert to 3a07 · already on disk 12. halted · per-path evidence · key diff A Configuration Change — Validated, Canaried, Halted, Reverted Region and global rings repeat the canary steps. A revert changes a pointer; the previous bundle never left the PoP. v 1.0 · owner Edge Platform · date 2026-09

Decisions

  • The halt condition includes hit ratio per path pattern, not only 5xx. A key change that fragments the cache returns no errors and still costs money and latency.
  • Revert is a pointer change to a bundle already on the PoP's disk. It does not wait for a download, a build or the control plane.
  • The rollout is a Temporal workflow, so a control-plane restart mid-rollout resumes the bake timer instead of forgetting the rollout.

Halt thresholds

  • 5xx rate up by more than 0.1 percentage points; hit ratio down by more than 2 points on any path pattern with at least 1% of the property's requests; stale-served ratio up by more than 1 point. Evaluated against the same PoP's previous week at the same hour.

Numbers

  • Push to a ring within 60 s p95; halt within 30 s of a regression being visible; revert within 60 s. The canary is WAW, an edge-only PoP with enough traffic to measure and few enough users to limit harm.
◀  Freshness Classes by Origin State All views Deployment  ▶
Solution Architecture v1.0 · Edge Architecture · 2026-09 · 29 views · open source, fully on-premises19-config-ring-rollout