Distributed Workflow Orchestration Platform  ·  View 20 of 31  ·  Runtime

Dead-Letter Capture and Operator Recovery

Where a task goes when retries are exhausted, and the five things a human can do about it.

Editable source SVG draw.io All views
Detect
Detect
Capture
Capture
Triage
Triage
Disposition
Disposition
Automatic
Automatic
Retries exhausted
application dead-letter
Retries exhausted...
Native DLQ
maxDelivery · TTL · poison
Native DLQ...
dlq-drain job
60 s poll · both sources
dlq-drain job...
Auto-classification
error code → cause group
Auto-classification...
Dead-letter store
Dead-letter store
dead_letter_record
full error + attempt history
dead_letter_record...
Payload snapshot
Blob · 180 d
Payload snapshot...
Searchable index
tenant · type · errorCode
Searchable index...
Operator
Operator
DLQ depth alert
> 50 in 15 min
DLQ depth alert...
Inspect timeline
events + traces + payload
Inspect timeline...
Retry task
same execution
Retry task...
Replay execution
new id · parent linked
Replay execution...
Discard
reason mandatory
Discard...
Terminate execution
compensations fire
Terminate execution...
Effect on state
Effect on state
New task_attempt
attempt count preserved
New task_attempt...
Execution FAILED
audited with actor
Execution FAILED...
Dead-Letter Capture and Operator Recovery
Dead-Letter Capture and Operator Recovery
Risk / gap
Risk / gap
Queue / topic
Queue / topic
Application we own
Application we own
Data store
Data store
Security / platform
Security / platform
Person or role
Person or role
failure / alternate
failure / alternate
synchronous
synchronous
event / async
event / async
Two failure sources, one queue for humans. Native Service Bus dead-letters are infrastructure problems and application dead-letters are business problems, but an operator should never have to know which console to open.
Two failure sources, one queue for humans. Native Service Bus dead-letters are infrastructure problems and application dead-letters are business problems, but an operator should never have to know which console to open.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

Decisions

  • Two failure sources, one queue for humans. Native Service Bus dead-letters and application dead-letters land in the same Cosmos-backed store with the same shape, so an operator never has to know which mechanism produced a failure.
  • The dead-letter record carries everything the requirement asks for and one thing it does not: the trace id. Without it, the record tells you what failed but not what the system was doing at the time.
  • Discard requires a reason and terminate fires the workflow's compensation branch if one exists. Destructive operator actions are audited with an actor, because a dead-letter queue is where data loss becomes deliberate.

Operator semantics

  • Retry replays the single task within the original execution, preserving the attempt history and adding a new task_attempt. The execution continues from where it stopped.
  • Replay creates a new execution linked to its parent, used when the original is too old or its version has been deprecated. It never mutates the original.
  • Alerting is on DLQ arrival rate, not depth alone — more than 50 in 15 minutes pages, because a slow trickle of dead-letters is a backlog while a burst is an incident.

Risks

  • Bulk retry of a large dead-letter backlog can re-saturate the downstream that caused it. The operator API rate-limits bulk actions and requires a confirmation above 100 items.
  • Payload snapshots may contain personal data and are retained 180 days. They inherit the workflow's data classification and are subject to deletion requests, which is an operational process rather than an automated one.
  • Event Grid dead-letters land in Blob and are not yet merged into this store. Known gap for V1.1, listed in view 07.