Distributed Workflow Orchestration Platform  ·  View 02 of 31  ·  Context and scope

High-Level Architecture

The path from a submitted execution to a completed one, in one picture.

Editable source SVG draw.io All views
Define
Define
Workflow Service
validate · version · publish
Workflow Service...
Admit
Admit
API Management
JWT · quota · throttle
API Management...
Execution API
idempotent submit
Execution API...
Decide
Decide
Orchestration Service
stateless · competing
Orchestration Service...
Scheduler Service
leader-elected
Scheduler Service...
Distribute
Distribute
Outbox Relay
change feed
Outbox Relay...
Azure Service Bus
task + completion queues
Azure Service Bus...
Execute
Execute
Worker Fleet
KEDA-scaled on AKS
Worker Fleet...
Task Handler Plugins
http · c360 · rag · notify
Task Handler Plugins...
Record
Record
Cosmos DB
source of truth
Cosmos DB...
Blob Storage
claim-check payloads
Blob Storage...
Observe
Observe
Azure Monitor
OpenTelemetry
Azure Monitor...
Operator API
retry · replay · cancel
Operator API...
change feed
change feed
TaskCompleted
TaskCompleted
advance DAG
advance DAG
intervene
intervene
High-Level Architecture — Submission to Completion
High-Level Architecture — Submission to Completion
Application we own
Application we own
Interface / broker
Interface / broker
Queue / topic
Queue / topic
Data store
Data store
Security / platform
Security / platform
event / async
event / async
synchronous
synchronous
One journey, one direction, with a single loop: results return through the bus so the orchestrator advances the DAG without holding a thread.
One journey, one direction, with a single loop: results return through the bus so the orchestrator advances the DAG without holding a thread.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

The shape of the argument

  • Seven stages, one direction, and exactly one loop. The loop matters: results return through the bus so the orchestrator advances the DAG on an event rather than holding a thread while a task runs.
  • The Outbox Relay sits between the database and the queue on purpose. Without it the orchestrator would have to write state and send a message as two operations, and there is no ordering of those two that is safe — view 13 walks through why.
  • Observe is a stage, not a sidebar. The correlation identifiers are minted at admission and carried through every message property, which is what makes view 25's execution waterfall possible at all.

Why these services

  • Service Bus over Event Hubs or Storage Queues: this is competing-consumer work distribution needing per-message locks, lock renewal for long tasks, scheduled enqueue for backoff, sessions for ordered completion and a native dead-letter queue. Event Hubs is a partitioned log with none of those.
  • Cosmos DB over Azure SQL for execution state: partition-scoped transactional batches, a change feed that makes the outbox and every projection possible without polling, elastic RU for a spiky workload, and a TTL that expires events without a purge job.
  • AKS over Container Apps or Durable Functions: the requirement explicitly asks to demonstrate pod disruption, node failure, HPA/KEDA behaviour and rolling deployments. Durable Functions would also make the runtime the orchestrator, which is precisely the coupling view 03 refuses.

Deliberate omissions

  • Failure paths are absent so the happy path stays readable. Views 16, 19, 20 and 31 carry them.
  • Backpressure, tenancy and security are not on this page; views 21, 29 and 27 own them.
  • The Retry Service and Trigger Service exist but are folded into Decide here. View 05 lists every deployable unit.