Distributed Workflow Orchestration Platform  ·  View 03 of 31  ·  Context and scope

Control, Transport and Execution Planes

The one principle every later view obeys: the database is the source of truth, and the queue is only how work travels.

Editable source SVG draw.io All views
Control plane — state is authoritative here
Control plane — state is authoritative here
Decision services
Decision services
Workflow Service
Workflow Service
Execution API
Execution API
Orchestration Service
Orchestration Service
Scheduler Service
Scheduler Service
Retry Service
Retry Service
System of record
System of record
Definitions
immutable versions
Definitions...
Execution state
ETag guarded
Execution state...
Execution events
append only
Execution events...
Leases
fencing tokens
Leases...
Transport plane — carries work, never truth
Transport plane — carries work, never truth
Service Bus namespace
Service Bus namespace
Task queues
by priority class
Task queues...
Completion queue
Completion queue
Scheduled messages
retry backoff
Scheduled messages...
Dead-letter queues
Dead-letter queues
Execution plane — stateless and replaceable
Execution plane — stateless and replaceable
Worker pools
Worker pools
Pool A — fast HTTP
p95 2s
Pool A — fast HTTP...
Pool B — enrichment
p95 40s
Pool B — enrichment...
Pool C — inference
GPU optional
Pool C — inference...
Downstream systems
HTTP · SaaS · LLM
Downstream systems...
transactional batch
transactional batch
outbox relay
outbox relay
at-least-once
at-least-once
side effect
side effect
TaskCompleted
TaskCompleted
advance DAG
advance DAG
Control, Transport and Execution Planes
Control, Transport and Execution Planes
Application we own
Application we own
Data store
Data store
Queue / topic
Queue / topic
External / third party
External / third party
synchronous
synchronous
event / async
event / async
A worker never writes execution state. It emits a result, and the control plane decides what that result means — which is exactly what makes a worker safe to kill mid-task.
A worker never writes execution state. It emits a result, and the control plane decides what that result means — which is exactly what makes a worker safe to kill mid-task.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

The decision that is the architecture

  • State lives in Cosmos DB. Service Bus carries pointers to work. The alternative — treating the queue as the state store, which is what a naive implementation drifts into — was rejected because you cannot query it, cannot audit it, cannot replay it, and cannot let an operator intervene in it.
  • Workers never write execution state. They emit a result and settle a message; the control plane decides what that result means. That single rule is why a worker can be killed at any instant without corrupting anything, and why view 24 can scale the fleet to zero.
  • The cost is honest: every task involves at least one queue hop and one database write more than a direct call would. That overhead buys recoverability, and at 900 tasks per second it is a Cosmos RU line item rather than an architectural problem.

What follows from it

  • Recovery after a total messaging outage is a replay from Cosmos of every task in SCHEDULED or RUNNING whose lease has expired. No message needs to survive (view 22).
  • Duplicate delivery becomes an expected event rather than a defect, because the state machine transition is idempotent under the ETag guard (views 16 and 17).
  • Orchestrator instances need no affinity to an execution. Any instance can advance any workflow, so the control plane scales and restarts freely (view 14).

Risks

  • Cosmos becomes the availability floor for the whole control plane. View 31 states the declared behaviour under throttling and regional loss, and view 22 prices the single-write-region choice.
  • The transport plane can drift into holding state if a future feature stores decisions in message properties. This is a design review checkpoint, not something a diagram prevents.
  • Per-task database round trips are the dominant RU cost. If the workload shifts to very short tasks, batching completions becomes necessary and is not designed for in V1.