Distributed Workflow Orchestration Platform  ·  View 14 of 31  ·  Runtime

DAG Dependency Resolution & Concurrent Fan-Out

How the orchestrator decides a task is ready, without any instance owning the execution.

Editable source SVG draw.io All views
Level 0 — root
Level 0 — root
create-customer
http · no dependencies
create-customer...
Level 1 — fan-out
Level 1 — fan-out
verify-identity
identity-check
verify-identity...
score-risk
customer360
score-risk...
fetch-credit-file
http · 6 s p95
fetch-credit-file...
Join gate
Join gate
All predecessors COMPLETED?
counter reaches zero
All predecessors COMPLETED?...
Tenant slot available?
Tenant slot available?
Deferred
re-evaluated on release
Deferred...
Level 2 — join
Level 2 — join
enrich-customer
customer360
enrich-customer...
Level 3
Level 3
send-notification
notification
send-notification...
Terminal
Terminal
Execution COMPLETED
output assembled
Execution COMPLETED...
Any task DEAD_LETTERED
execution FAILED
Any task DEAD_LETTERED...
ready
ready
at limit
at limit
on failure
on failure
DAG Dependency Resolution & Concurrent Fan-Out
DAG Dependency Resolution & Concurrent Fan-Out
Application we own
Application we own
Decision point
Decision point
Risk / gap
Risk / gap
synchronous
synchronous
failure / alternate
failure / alternate
Readiness is a pending-predecessor counter on each task document, decremented inside the same ETag-guarded write that records the predecessor's completion. Two orchestrators racing the same completion cannot both dispatch the successor.
Readiness is a pending-predecessor counter on each task document, decremented inside the same ETag-guarded write that records the predecessor's completion. Two orchestrators racing the same completion cannot both dispatch the successor.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

Decisions

  • Readiness is a pending-predecessor counter on each task document, decremented inside the same ETag-guarded write that records the predecessor's completion. Two orchestrators racing the same completion cannot both dispatch the successor.
  • Orchestration is stateless and competing rather than one long-lived actor per execution. Any instance can advance any workflow, which is what lets the control plane restart, scale and deploy without draining executions.
  • A ready task that cannot start because the tenant is at its concurrency limit is deferred, not failed. Deferral is a state (WAITING_SLOT), so it is visible, alertable and fair rather than an invisible queue delay.

Concurrency semantics

  • Level-1 tasks are dispatched together; the platform makes no ordering promise among siblings. A workflow that needs ordering must express it as an edge, not assume it.
  • The join gate fires exactly once per successor, guaranteed by the counter transition rather than by a query for 'are all predecessors done', which would race.
  • Fan-out is bounded by the workflow's 500-task cap and by the tenant's concurrency lease, so a single execution cannot consume a tenant's whole allocation.

Risks

  • A deferred task depends on a slot-release signal. If that signal is lost, the task waits until a reconciliation sweep finds it — a 60-second sweep exists precisely because event-driven-only would be a liveness bug.
  • Very wide fan-out concentrates writes on one logical partition and can throttle at 10,000 RU/s. Above roughly 200 concurrent siblings this becomes measurable and is a load-test target.
  • Conditional branches and dynamic fan-out over a collection are not in V1. Every DAG is static once bound, which is a real functional limitation to confirm with users.