Distributed Workflow Orchestration Platform  ·  View 13 of 31  ·  Runtime

Submission, Persistence and First Dispatch

The critical flow, end to end, including where the client is answered and why that is safe.

Editable source SVG draw.io All views
Client application
Client application
API Management
API Management
Execution API
Execution API
Cosmos DB
Cosmos DB
Outbox Relay
Outbox Relay
Service Bus
Service Bus
Worker
Worker
Orchestration Service
Orchestration Service
1. POST /workflows/customer-onboarding/executions
1. POST /workflows/customer-onboarding/executions
2. validate JWT · tenant quota · rate limit
2. validate JWT · tenant quota · rate limit
3. forward with tenantId claim
3. forward with tenantId claim
4. read pinned workflow version
4. read pinned workflow version
5. check Idempotency-Key header
5. check Idempotency-Key header
6. batch: execution PENDING + root tasks + event + outbox
6. batch: execution PENDING + root tasks + event + outbox
7. committed · _etag
7. committed · _etag
8. 202 Accepted · executionId
8. 202 Accepted · executionId
9. change feed · pending outbox rows
9. change feed · pending outbox rows
10. send task message · MessageId = taskExecutionId
10. send task message · MessageId = taskExecutionId
11. mark published_at
11. mark published_at
12. peek-lock deliver
12. peek-lock deliver
13. idempotency claim · execute handler
13. idempotency claim · execute handler
14. TaskCompleted to completions queue
14. TaskCompleted to completions queue
15. session-ordered completion
15. session-ordered completion
16. transition · dispatch ready successors
16. transition · dispatch ready successors
Critical Flow — Submission, Persistence and First Dispatch
Critical Flow — Submission, Persistence and First Dispatch
The client is answered at step 8, before any task runs. Everything after that point is recoverable from Cosmos alone, which is what makes the API able to promise 202 in under 300 ms.
The client is answered at step 8, before any task runs. Everything after that point is recoverable from Cosmos alone, which is what makes the API able to promise 202 in under 300 ms.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

Decisions

  • The client is answered at step 8, before any task has run. Everything after that point is recoverable from Cosmos alone, so a 202 is a durable promise rather than an optimistic one.
  • The Idempotency-Key header makes submission itself idempotent. A client retrying a timed-out POST gets the same executionId rather than a second execution — the requirement's idempotency rule applied at the front door as well as at the worker.
  • The dispatch message carries the taskExecutionId as its MessageId, which lets Service Bus duplicate detection suppress relay-level double sends within a 10-minute window without anyone claiming exactly-once.

Latency budget

  • p95 under 300 ms for steps 1 to 8: roughly 30 ms at API Management, 40 ms for the version read, 60 ms for the batch write, the rest network and serialisation.
  • p95 under 1 second from commit to a message on the queue, which is change-feed latency plus relay processing. This is the number that KEDA's oldest-message-age trigger in view 24 is scaled against.
  • The version read is a point read inside the tenant partition and is cached per pod for 60 seconds, because an immutable version is safe to cache indefinitely.

Failure behaviour

  • If the batch write fails, the client gets 503 and nothing exists. This is the only clean failure point in the flow, which is why it is placed before the response.
  • If the relay dies between send and marking published_at, the message is sent twice. Duplicate detection catches it inside 10 minutes; the worker's idempotency catches it after that (view 17).
  • If Service Bus is unavailable, submissions still succeed and outbox rows accumulate. Dispatch resumes on recovery with no data loss and a visible lag metric.