# Distributed Workflow Orchestration Platform

**Solution Architecture v1.0 · Data & AI Global Practice · 2026-08 · 31 views · Microsoft Azure**

An Azure-native platform that executes DAG workflows across a horizontally scalable worker fleet, with durable state, at-least-once task delivery, application-level idempotency, configurable retries, dead-letter recovery and per-tenant backpressure. Six decisions carry the design: Cosmos DB is the source of truth and Service Bus only distributes work; a transactional outbox replaces the dual write that would otherwise lose or invent tasks; orchestration is stateless and competing rather than one long-lived actor per execution; retries are re-enqueued as scheduled messages instead of abandoned onto the queue, so backoff and jitter are real; the scheduler is the only leader-elected component and it is fenced with a monotonic token rather than trusted with a lease; and at-least-once is stated as the contract, with idempotency pushed to the task handler and its downstream.

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The view index — 31 views grouped into six acts, every format linked |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning it deliberately omits, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same 31 views as SVG with the diagram XML embedded — re-opens in diagrams.net fully editable |
| `diagrams/drawio/*.drawio` | draw.io native source |
| `specs/views.json` | Diagram specifications — the source of truth for every view (assembled from `part-a..f.json`) |
| `specs/manifest.json` | Acts, page titles, subtitles and the decision / assumption / risk cards (assembled from `manifest-a..b.json`) |
| `scripts/build.sh` | Rebuilds every deliverable from the specs |
| `scripts/pin-icons.mjs` | Pins one mark per component so the same box carries the same icon on every page |
| `ask.md` | The original requirement |

Every component carries its own mark — 479 icons embedded in the files, so the deliverable renders identically anywhere with no external requests. Each view page links to its own SVG and draw.io source, back to the index, and to the previous and next view.

To rebuild after editing a spec:

```bash
bash scripts/build.sh          # requires Node 20+ and draw.io Desktop
```

Edit `specs/part-a.json` … `part-f.json` and `specs/manifest-a.json` / `manifest-b.json`; the build script assembles `views.json` and `manifest.json` from them.

---

## The six acts

| Act | What it lands | Views |
|---|---|---|
| 1 · Context and scope | The boundary, the trigger sources, and the plane separation that governs everything after it | 01–03 |
| 2 · Structure | The layering rule, the components, every interface, the messaging topology, and how a DAG becomes an immutable version | 04–08 |
| 3 · Data | Where execution state lives, the entities a decision reads, how data moves with its cadence, and what the event log is for | 09–12 |
| 4 · Runtime | Submission, dependency resolution, triggering, the task state machine, duplicate delivery, the worker pipeline, retries, dead-lettering, backpressure | 13–21 |
| 5 · Operations | Where it runs, how a change reaches production, how the fleet scales on queue pressure, what is watched, and the operator's loop | 22–26 |
| 6 · Assurance | Trust boundaries, the secretless token path, four layers of tenant isolation, split-brain prevention, and every failure mode | 27–31 |

---

## The 31 views

| # | View | Layout | What it answers |
|---|---|---|---|
| 01 | System Context | context | Who triggers workflows, what the tasks act on, what the platform never owns |
| 02 | High-Level Architecture | flow | The path from a submitted execution to a completed one |
| 03 | Control, Transport & Execution Planes | nested | Why the database is the source of truth and the queue only carries work |
| 04 | Layered Architecture | bands | What depends on what, and the one dependency pointing the wrong way |
| 05 | Container & Component Architecture | nested | Every deployable unit, its replica count, and its managed dependency |
| 06 | Integration & Interface Catalogue | hub | Every way something enters or leaves, with protocol and cadence |
| 07 | Messaging Topology & Queue Classes | nested | Which queues exist, why several, and how a failure reaches a human |
| 08 | Definition Lifecycle | flow | How a DAG becomes an immutable published version, and what binding protects |
| 09 | Data Architecture & Storage Zones | nested | What is authoritative, what is rebuildable, what backup must cover |
| 10 | Execution & Metadata Data Model | er | The thirteen entities an orchestration decision reads or writes |
| 11 | Execution Data Flow | flow | Where execution data comes from and goes, at what cadence |
| 12 | Event Log & State Projection | swimlane | What the event log is for, and what it is deliberately not used for |
| 13 | Submission, Persistence & First Dispatch | sequence | The critical flow, and why the client can be answered before any task runs |
| 14 | DAG Dependency Resolution | flow | How the orchestrator decides a task is ready, without owning the execution |
| 15 | Trigger Modes | swimlane | Four ways an execution starts, and the single path they converge on |
| 16 | Task State Machine | flow | Every state, every legal transition, and the three that are terminal |
| 17 | At-Least-Once Delivery & Idempotency | sequence | The duplicate that will happen, and what stops a duplicate side effect |
| 18 | Worker Runtime Pipeline | flow | What happens between a message arriving and a message being settled |
| 19 | Retry, Backoff & Circuit Breaking | flow | How a failure becomes another attempt, and when it stops trying |
| 20 | Dead-Letter Capture & Operator Recovery | swimlane | Where an exhausted task goes, and the five things a human can do |
| 21 | Backpressure & Concurrency Governance | grid | Five control scopes, four enforcement points, one place work is refused |
| 22 | Deployment & Infrastructure | nested | What runs where, which failures it survives, what recovery was bought |
| 23 | CI/CD and Environments | flow | How a change reaches production, and every gate that stops a bad one |
| 24 | Queue-Driven Autoscaling with KEDA | flow | What the fleet scales on, and where scaling stops helping |
| 25 | Observability & Distributed Tracing | grid | The signals, their destinations, and why execution 123 took 47 seconds |
| 26 | Operations Lifecycle Loop | cycle | The loop an operator works in, and the edge that closes it |
| 27 | Security Architecture — Trust Zones | zones | Where the boundaries are and where an attacker would arrive |
| 28 | Secretless Access — Token Path | sequence | How a pod reaches a database without holding a credential |
| 29 | Multi-Tenant Isolation Model | nested | Four layers of isolation, and the parts that are honestly shared |
| 30 | Leader Election & Fencing | sequence | Why a lease alone does not prevent split brain |
| 31 | Failure Modes & Residual Risk | nested | Twenty-one failure modes, their declared behaviour, and the five not solved |

---

## Stated assumptions

None of these came from a client; they are exercise assumptions chosen to make the design specific, and they are the first thing to confirm against real traffic.

- 40 tenants, 2,400 published workflow definitions, average workflow 12 tasks with a 500-task ceiling.
- 1.8M task executions per day, peaking near 900 per second; 10,000 concurrently running executions.
- p99 task payload 40 KB; anything above 64 KB travels by reference through Blob Storage.
- Engineering targets: 99.9% API availability, p95 submit under 300 ms, p95 task scheduling under 1 second, worker fleet elastic from 1 to 100+ pods.
- Single write region with zone redundancy: RTO 4 hours, RPO 5 minutes.

## What this design does not do

- **Exactly-once processing.** At-least-once delivery plus application idempotency is the contract, stated on view 17 rather than implied.
- **Multi-region active-active writes.** Conflict resolution on a state machine is a correctness problem, not a merge problem; deferred with the cost stated on view 22.
- **Conditional branches and dynamic fan-out.** Every DAG is static once bound (view 14) — a real functional limitation worth confirming with users.
- **Compensating transactions.** Saga semantics are a workflow-authoring pattern, not a platform state; there is no automatic rollback.
- **Per-tenant control-plane isolation.** The control plane, cluster and Cosmos RU pool are shared, drawn in red on view 29.