Distributed Workflow Orchestration Platform
An Azure-native platform that executes DAG workflows across a horizontally scalable worker fleet, with durable state, at-least-once task delivery, application-level idempotency, configurable retries, dead-letter recovery and per-tenant backpressure. Six decisions carry the design: Cosmos DB is the source of truth and Service Bus only distributes work; a transactional outbox replaces the dual write that would otherwise lose or invent tasks; orchestration is stateless and competing rather than one long-lived actor per execution; retries are re-enqueued as scheduled messages instead of abandoned onto the queue, so backoff and jitter are real; the scheduler is the only leader-elected component and it is fenced with a monotonic token rather than trusted with a lease; and at-least-once is stated as the contract, with idempotency pushed to the task handler and its downstream.
31 views, each in three formats.
Open a view to read it in full. Every SVG carries its diagram source inside it, so it opens in diagrams.net fully editable with no import step; the draw.io files are the same diagrams as plain source.
-
01
System Context
Who triggers workflows, what the platform's tasks act on, and what it deliberately never owns.
-
02
High-Level Architecture
The path from a submitted execution to a completed one, in one picture.
-
03
Control, Transport and Execution Planes
The one principle every later view obeys: the database is the source of truth, and the queue is only how work travels.
-
04
Layered Architecture
What depends on what, and the one dependency that points the wrong way.
-
05
Container & Component Architecture
Every deployable unit, its replica count, and the managed service it depends on.
-
06
Integration & Interface Catalogue
Every way something enters or leaves the platform, with protocol, direction and cadence.
-
07
Messaging Topology & Queue Classes
Which queues exist, why there are several, and how a failed message reaches a human.
-
08
Workflow Definition — Validation, Versioning and Binding
How a DAG becomes an immutable published version, and what a running execution is protected from.
-
09
Data Architecture & Storage Zones
What is authoritative, what is rebuildable, and what the backup plan therefore has to cover.
-
10
Execution & Metadata Data Model
The thirteen entities an orchestration decision reads or writes.
-
11
Execution Data Flow
Where execution data comes from, what happens to it on the way, and at what cadence.
-
12
Execution Event Log & State Projection
What the event log is for, and what it is deliberately not used for.
-
13
Submission, Persistence and First Dispatch
The critical flow, end to end, including where the client is answered and why that is safe.
-
14
DAG Dependency Resolution & Concurrent Fan-Out
How the orchestrator decides a task is ready, without any instance owning the execution.
-
15
Trigger Modes — Immediate, Scheduled, Event-Driven and Replay
Four ways an execution starts, and the single path they all converge on.
-
16
Task State Machine
Every state a task can be in, every legal transition, and the three that are terminal.
-
17
At-Least-Once Delivery & Idempotency
The duplicate the platform promises will happen, and what stops it becoming a duplicate side effect.
-
18
Worker Runtime Execution Pipeline
What happens between a message arriving and a message being settled.
-
19
Retry, Backoff & Circuit Breaking
How a failure becomes another attempt, and how a failing dependency stops consuming the retry budget.
-
20
Dead-Letter Capture and Operator Recovery
Where a task goes when retries are exhausted, and the five things a human can do about it.
-
21
Backpressure & Concurrency Governance
Five control scopes against four enforcement points, and the one place a request is actually refused.
-
22
Deployment & Infrastructure
What runs where, which failures it survives, and the recovery posture that was actually bought.
-
23
CI/CD and Environments
How a change reaches production, and every gate that can stop a bad one.
-
24
Queue-Driven Autoscaling with KEDA
What the fleet scales on, why it is not queue depth, and where scaling stops helping.
-
25
Observability and Distributed Tracing
The signals, where they go, and the specific question the design has to be able to answer.
-
26
Operations Lifecycle Loop
The loop an operator actually works in, and the edge that makes it a lifecycle rather than an escalation path.
-
27
Security Architecture — Trust Zones
Where the boundaries are, what authenticates at each crossing, and where an attacker would actually arrive.
-
28
Secretless Access — Token Path
How a caller proves who it is, and how a pod reaches a database without holding a credential.
-
29
Multi-Tenant Isolation Model
Four layers of isolation, what each one costs, and the parts that are honestly shared.
-
30
Leader Election and Fencing
The one component that needs a single active coordinator, and why a lease alone is not enough.
-
31
Failure Modes and Accepted Residual Risk
Twenty-one ways this platform fails, what it does about each, and the five it chose not to solve.
Everything as it was delivered.
These files are served exactly as they were produced — the diagram pages keep their own house style because that is the artifact, not a rendering of it.