Distributed Workflow Orchestration Platform  ·  View 23 of 31  ·  Operations

CI/CD and Environments

How a change reaches production, and every gate that can stop a bad one.

Editable source SVG draw.io All views
Source
Source
GitHub monorepo
services · workers · IaC
GitHub monorepo...
Pull request
2 reviewers · CODEOWNERS
Pull request...
Build
Build
GitHub Actions
OIDC to Azure · no PAT
GitHub Actions...
Unit + mutation tests
80% line gate
Unit + mutation tests...
Container build
distroless · SBOM
Container build...
Verify
Verify
Integration tests
Cosmos + SB emulators
Integration tests...
Contract tests
task + workflow events
Contract tests...
Security scan
Defender · Trivy · secrets
Security scan...
Terraform plan
policy-as-code gate
Terraform plan...
Package
Package
Push to ACR
immutable digest tag
Push to ACR...
Sign + attest
Notation · provenance
Sign + attest...
Helm chart to OCI
versioned with image
Helm chart to OCI...
Promote
Promote
dev
auto on merge
dev...
staging
load + chaos suite
staging...
Change approval
manual · change record
Change approval...
Production
Production
Rolling control plane
PDB · maxUnavailable 1
Rolling control plane...
Worker drain-and-replace
finish lease, then exit
Worker drain-and-replace...
Auto rollback
SLO burn > 2% in 15 min
Auto rollback...
on merge
on merge
previous digest
previous digest
CI/CD — How a Change Reaches Production, and What Stops a Bad One
CI/CD — How a Change Reaches Production, and What Stops a Bad One
External / third party
External / third party
Decision point
Decision point
Application we own
Application we own
Security / platform
Security / platform
Risk / gap
Risk / gap
synchronous
synchronous
failure / alternate
failure / alternate
Zero-downtime rests on one worker behaviour: on SIGTERM a worker stops receiving, finishes or abandons its in-flight lease, then exits. terminationGracePeriodSeconds is set above the longest task's lock renewal window.
Zero-downtime rests on one worker behaviour: on SIGTERM a worker stops receiving, finishes or abandons its in-flight lease, then exits. terminationGracePeriodSeconds is set above the longest task's lock renewal window.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

Decisions

  • Gates, not stages. Each step on this page can fail the pipeline: coverage, integration, contract, security scan, Terraform policy, and a human change approval before production.
  • Contract tests on task and workflow events are non-negotiable, because handlers are deployed independently of the orchestrator. A message schema change that passes unit tests and breaks a worker is the failure mode this exists to prevent.
  • GitHub Actions authenticates to Azure with OIDC federation. No service principal secret exists in the CI system, which removes the most commonly leaked credential in an enterprise estate.

Zero-downtime mechanics

  • Control plane rolls with a PodDisruptionBudget and maxUnavailable of 1. In-flight HTTP requests drain; nothing else is in flight because the control plane holds no long operations.
  • Workers drain rather than roll: on SIGTERM a pod stops receiving, finishes or abandons its lease, then exits. A task interrupted mid-flight is redelivered, which is safe precisely because of view 17.
  • Automatic rollback triggers on SLO burn above 2% in 15 minutes, redeploying the previous image digest. Rollback is to a digest, never a tag, because tags move.

Assumptions

  • Staging carries a load and chaos suite that reproduces every failure mode in view 31. A release that has not run it is not a release candidate.
  • Infrastructure is Terraform with policy-as-code; Helm charts are versioned with the image they deploy so a rollback moves both together.
  • Database schema changes are additive and forward-compatible for one release, so a rollback never requires a data migration.