Distributed Workflow Orchestration Platform  ·  View 18 of 31  ·  Runtime

Worker Runtime Execution Pipeline

What happens between a message arriving and a message being settled.

Editable source SVG draw.io All views
Receive
Receive
Peek-lock receive
prefetch 32
Peek-lock receive...
Envelope validation
schema + version
Envelope validation...
Guard
Guard
Cancelled or superseded?
Cancelled or superseded?
Idempotency claim
taskExecutionId
Idempotency claim...
Local semaphore
per type · per tenant
Local semaphore...
Prepare
Prepare
Resolve handler
plugin registry
Resolve handler...
Hydrate payload
claim-check from Blob
Hydrate payload...
Acquire credential
workload identity
Acquire credential...
Execute
Execute
execute(context)
handler contract
execute(context)...
Lock auto-renewal
background · to 2 h
Lock auto-renewal...
Timeout watchdog
per-task deadline
Timeout watchdog...
Record
Record
Persist output
Blob if > 64 KB
Persist output...
Idempotency COMPLETED
Idempotency COMPLETED
Emit TaskCompleted
completions queue
Emit TaskCompleted...
Settle
Settle
Complete message
happy path
Complete message...
Close OTel span
trace + task attributes
Close OTel span...
Abandon
transient · immediate redeliver
Abandon...
Dead-letter
poison envelope only
Dead-letter...
transient error
transient error
unparseable
unparseable
Worker Runtime — From Message to Settlement
Worker Runtime — From Message to Settlement
Queue / topic
Queue / topic
Application we own
Application we own
Decision point
Decision point
Data store
Data store
Security / platform
Security / platform
Risk / gap
Risk / gap
failure / alternate
failure / alternate
The runtime does everything except decide what a failure means. It reports the outcome and settles the message; the retry policy and state transition are the control plane's decision, made from the workflow definition.
The runtime does everything except decide what a failure means. It reports the outcome and settles the message; the retry policy and state transition are the control plane's decision, made from the workflow definition.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

Decisions

  • The runtime does everything except decide what a failure means. It reports an outcome and settles the message; retry policy and state transition are control-plane decisions read from the workflow definition, which is what keeps retry behaviour a property of the workflow rather than of the worker image.
  • Cancellation is checked at receipt, before the idempotency claim. A cancelled execution's in-flight messages are dropped and settled rather than executed, so cancellation is fast without needing to reach into the queue.
  • Credentials are acquired per execution from the pod's workload identity, never read from environment or config. A handler receives a token with the scope it needs and nothing broader (view 28).

Timing and limits

  • Lock auto-renewal runs on a background timer and extends to a 2-hour ceiling for the long-running class. Renewal failure is treated as lease loss, and the handler is cancelled rather than allowed to continue against an expired lock.
  • The timeout watchdog enforces the per-task deadline independently of the lock, because a handler that hangs on a socket will not notice either one on its own.
  • Local semaphores bound in-process concurrency per type and per tenant, which is the last of the four backpressure points in view 21 and the only one that protects the pod itself.

Settlement rules

  • Abandon is used only for transient errors where an immediate redelivery is wanted. Everything else is completed with an emitted failure result, so the control plane owns the retry decision.
  • Native dead-lettering by the worker is reserved for unparseable envelopes. A business failure never dead-letters at the message layer, because the error context would be lost.
  • On SIGTERM the worker stops receiving, finishes or abandons its in-flight lease, then exits. terminationGracePeriodSeconds is set above the longest lock renewal window, which is what makes view 23's zero-downtime claim true.