Spot Capacity Orchestrator  ·  View 26 of 32  ·  6 · Operations

Change Pipeline

Contract changes and code changes take different routes to the same canary, and neither can reach the whole fleet in one step.

Editable source SVG draw.io All views
Propose Contract change Forgejo pull request Agent or controller change Go · pull request Check Schema and feasibility CUE · 30-day replay Floor or spend changed? Authorise Governance signature second key Build and sign cosign · image bake Canary One partition bundle · 24 h New launches only 5% of pools Gate Canary clean? overruns · waste · alarms Fleet All partitions bundle pushed Fleet replaced by churn ~ 72 h to 95% Change — Contracts and Code Both Reach the Fleet Through a Canary Data store Application we own Decision point Security / platform Opportunity failure / alternate A dirty canary reverts the bundle or stops launching the new image. Agents are never upgraded in place on a draining node. The fleet turns over by reclamation and scale-down, which is the rollout. v 1.0 · owner Compute Platform Architecture · date 2026-09

Contracts

  • A contract pull request is checked against the schema and the feasibility rules, and replayed against 30 days of pool history to show how the workload would have been placed and how often it would have fallen back.
  • A change to a floor, a spend cap or fallback policy needs a governance signature from a separate group, on top of the normal review. The distributor refuses a bundle without it.
  • A signed bundle goes to one partition for 24 hours before the rest.

Code

  • Controllers deploy as a canary partition. The agent is not upgraded in place: a new image is used for 5% of launches, then all launches, and the fleet turns over through reclamation and scale-down. At 4% an hour, about 95% of the fleet is on the new agent within 72 hours without anyone draining a node to upgrade it.
  • Security fixes that cannot wait use an in-place restart on nodes that are not draining.

Gate

  • The canary fails on overrun rate above 0.5%, wasted work above 3% or any breach alarm the baseline partition did not also raise.
◀  Deployment Architecture All views Observability  ▶
Solution Architecture v1.0 · Compute Platform Architecture · 2026-09 · 32 views · open source, fully on-premises26-release-pipeline