Distributed Workflow Orchestration Platform

Architecture Views

31 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

An Azure-native platform that executes DAG workflows across a horizontally scalable worker fleet, with durable state, at-least-once task delivery, application-level idempotency, configurable retries, dead-letter recovery and per-tenant backpressure. The set reads in six acts: what sits inside the boundary, how the parts fit, where execution state lives, what happens at runtime, how it is operated, and why it is safe. Six decisions carry the whole design — Cosmos DB is the source of truth and Service Bus only distributes work (view 03); a transactional outbox replaces the dual write that would otherwise lose or invent tasks (views 11 and 13); orchestration is stateless and competing rather than one long-lived actor per execution (view 14); retries are re-enqueued as scheduled messages instead of abandoned onto the queue, so backoff and jitter are real (view 19); the scheduler is the only leader-elected component and it is fenced with a monotonic token rather than trusted with a lease (view 30); and at-least-once is stated as the contract, with idempotency pushed to the task handler and its downstream (view 17).

Context and scope

What the platform is accountable for, who triggers it, what its tasks reach out to, and the plane separation that governs every later view.

01 System Context Who triggers workflows, what the platform's tasks act on, and what it deliberately never owns. HTML page SVG draw.io

02 High-Level Architecture The path from a submitted execution to a completed one, in one picture. HTML page SVG draw.io

03 Control, Transport and Execution Planes The one principle every later view obeys: the database is the source of truth, and the queue is only how work travels. HTML page SVG draw.io

Structure

The layering rule, the deployable components, every interface, the messaging topology, and how a workflow definition becomes an immutable, bound version.

04 Layered Architecture What depends on what, and the one dependency that points the wrong way. HTML page SVG draw.io

05 Container & Component Architecture Every deployable unit, its replica count, and the managed service it depends on. HTML page SVG draw.io

06 Integration & Interface Catalogue Every way something enters or leaves the platform, with protocol, direction and cadence. HTML page SVG draw.io

07 Messaging Topology & Queue Classes Which queues exist, why there are several, and how a failed message reaches a human. HTML page SVG draw.io

08 Workflow Definition — Validation, Versioning and Binding How a DAG becomes an immutable published version, and what a running execution is protected from. HTML page SVG draw.io

Data

Where execution state lives and who owns it, the entities an orchestration decision reads, how data moves with its cadence, and what the event log is actually for.

09 Data Architecture & Storage Zones What is authoritative, what is rebuildable, and what the backup plan therefore has to cover. HTML page SVG draw.io

10 Execution & Metadata Data Model The thirteen entities an orchestration decision reads or writes. HTML page SVG draw.io

11 Execution Data Flow Where execution data comes from, what happens to it on the way, and at what cadence. HTML page SVG draw.io

12 Execution Event Log & State Projection What the event log is for, and what it is deliberately not used for. HTML page SVG draw.io

Runtime

The nine flows that are the platform: submission, dependency resolution, triggering, the task state machine, duplicate delivery, the worker pipeline, retries, dead-lettering and backpressure.

13 Submission, Persistence and First Dispatch The critical flow, end to end, including where the client is answered and why that is safe. HTML page SVG draw.io

14 DAG Dependency Resolution & Concurrent Fan-Out How the orchestrator decides a task is ready, without any instance owning the execution. HTML page SVG draw.io

15 Trigger Modes — Immediate, Scheduled, Event-Driven and Replay Four ways an execution starts, and the single path they all converge on. HTML page SVG draw.io

16 Task State Machine Every state a task can be in, every legal transition, and the three that are terminal. HTML page SVG draw.io

17 At-Least-Once Delivery & Idempotency The duplicate the platform promises will happen, and what stops it becoming a duplicate side effect. HTML page SVG draw.io

18 Worker Runtime Execution Pipeline What happens between a message arriving and a message being settled. HTML page SVG draw.io

19 Retry, Backoff & Circuit Breaking How a failure becomes another attempt, and how a failing dependency stops consuming the retry budget. HTML page SVG draw.io

20 Dead-Letter Capture and Operator Recovery Where a task goes when retries are exhausted, and the five things a human can do about it. HTML page SVG draw.io

21 Backpressure & Concurrency Governance Five control scopes against four enforcement points, and the one place a request is actually refused. HTML page SVG draw.io

Operations

Where it runs, how a change reaches production safely, how the fleet scales on queue pressure, what is watched, and the loop an operator actually works in.

22 Deployment & Infrastructure What runs where, which failures it survives, and the recovery posture that was actually bought. HTML page SVG draw.io

23 CI/CD and Environments How a change reaches production, and every gate that can stop a bad one. HTML page SVG draw.io

24 Queue-Driven Autoscaling with KEDA What the fleet scales on, why it is not queue depth, and where scaling stops helping. HTML page SVG draw.io

25 Observability and Distributed Tracing The signals, where they go, and the specific question the design has to be able to answer. HTML page SVG draw.io

26 Operations Lifecycle Loop The loop an operator actually works in, and the edge that makes it a lifecycle rather than an escalation path. HTML page SVG draw.io

Assurance

The trust boundaries and what crosses them, the secretless token path, four layers of tenant isolation with the shared parts named, split-brain prevention, and every failure mode with its declared behaviour.

27 Security Architecture — Trust Zones Where the boundaries are, what authenticates at each crossing, and where an attacker would actually arrive. HTML page SVG draw.io

28 Secretless Access — Token Path How a caller proves who it is, and how a pod reaches a database without holding a credential. HTML page SVG draw.io

29 Multi-Tenant Isolation Model Four layers of isolation, what each one costs, and the parts that are honestly shared. HTML page SVG draw.io

30 Leader Election and Fencing The one component that needs a single active coordinator, and why a lease alone is not enough. HTML page SVG draw.io

31 Failure Modes and Accepted Residual Risk Twenty-one ways this platform fails, what it does about each, and the five it chose not to solve. HTML page SVG draw.io

Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.