Distributed Workflow Orchestration Platform  ·  View 11 of 31  ·  Data

Execution Data Flow

Where execution data comes from, what happens to it on the way, and at what cadence.

Editable source SVG draw.io All views
Origin
Origin
Client submit
REST · sync
Client submit...
Schedule fire
cron · 1 min tick
Schedule fire...
Domain event
CloudEvents · async
Domain event...
Admit
Admit
Execution API
validate · authorise
Execution API...
Claim-check split
> 64 KB to Blob
Claim-check split...
Persist
Persist
Transactional batch
state + event + outbox
Transactional batch...
executions container
source of truth
executions container...
Distribute
Distribute
Change feed
ordered per partition
Change feed...
Service Bus
task queues
Service Bus...
Produce
Produce
Worker
handler executes
Worker...
Result + output ref
completion message
Result + output ref...
Project
Project
Projection workers
status · timeline · metrics
Projection workers...
Reporting load
micro-batch · 5 min
Reporting load...
Consume
Consume
Operations console
SSE · sub-second
Operations console...
Data warehouse
nightly extract
Data warehouse...
Azure Monitor
OTLP continuous
Azure Monitor...
payload ref only
payload ref only
state transition
state transition
change feed
change feed
nightly
nightly
Execution Data Flow — Origin to Consumption
Execution Data Flow — Origin to Consumption
External / third party
External / third party
Application we own
Application we own
Queue / topic
Queue / topic
Data store
Data store
Security / platform
Security / platform
synchronous
synchronous
event / async
event / async
batch
batch
Cadence is on every arrow because batch and streaming otherwise look identical. Payloads travel by reference from the moment they exceed 64 KB, so message size stays bounded regardless of workload.
Cadence is on every arrow because batch and streaming otherwise look identical. Payloads travel by reference from the moment they exceed 64 KB, so message size stays bounded regardless of workload.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

The write that has to be atomic

  • State, event and outbox row commit in one Cosmos transactional batch. This is the alternative to a dual write, and it is the difference between a platform that occasionally loses tasks and one that does not.
  • Write-then-send loses the task if the process dies between the two operations. Send-then-write invents a task that no state knows about. Neither failure is acceptable at 1.8M tasks per day, and neither can be fixed with a retry.
  • The change feed is the only dispatcher. It is checkpointed, ordered per partition and resumable, so an Outbox Relay crash delays dispatch rather than dropping it.

Cadence, stated everywhere

  • Submission is synchronous and answered in under 300 ms at p95. Dispatch is asynchronous and targeted at under 1 second at p95 from commit to queue.
  • Projections are near-real-time via the change feed; reporting is a 5-minute micro-batch; the warehouse extract is nightly. Three different cadences on one page, because conflating them is how capacity planning goes wrong.
  • Payloads move by reference above 64 KB in both directions — input at admission, output at completion — so message size is independent of workload shape.

Deliberate omissions

  • Failure and retry paths are omitted so cadence stays legible; views 19 and 20 carry them.
  • Per-tenant data segregation is not shown here; view 29 covers it at all four layers.
  • Encryption in transit and at rest applies to every arrow and store on this page and is documented once, on view 27.