Distributed Workflow Orchestration Platform  ·  View 19 of 31  ·  Runtime

Retry, Backoff & Circuit Breaking

How a failure becomes another attempt, and how a failing dependency stops consuming the retry budget.

Editable source SVG draw.io All views
Failure
Failure
Task attempt failed
exception · 5xx · timeout
Task attempt failed...
task_attempt written
error code + trace id
task_attempt written...
Classify
Classify
Error taxonomy
transient · throttled · permanent
Error taxonomy...
Retryable?
Retryable?
Budget
Budget
Retry policy
max 5 · exp · 2 s → 5 m
Retry policy...
Attempts remaining?
Attempts remaining?
Backoff
Backoff
delay = min(2s·2ⁿ, 5m)
full jitter applied
delay = min(2s·2ⁿ, 5m)...
Scheduled message
Service Bus enqueue time
Scheduled message...
Circuit
Circuit
Handler circuit state
Redis · per task type
Handler circuit state...
Circuit open?
Circuit open?
Half-open probe
1 in flight
Half-open probe...
Outcome
Outcome
Re-dispatched
attempt n+1
Re-dispatched...
Held, not failed
circuit open · no budget burn
Held, not failed...
Dead-lettered
operator owned
Dead-lettered...
permanent
permanent
exhausted
exhausted
open
open
closed
closed
Retry, Backoff and Circuit Breaking
Retry, Backoff and Circuit Breaking
Risk / gap
Risk / gap
Data store
Data store
Application we own
Application we own
Decision point
Decision point
Queue / topic
Queue / topic
failure / alternate
failure / alternate
synchronous
synchronous
Retries are re-enqueued as new scheduled messages rather than abandoned back onto the queue. Backoff can then exceed the lock duration, jitter is real, and the attempt count lives in state where the workflow definition can govern it.
Retries are re-enqueued as new scheduled messages rather than abandoned back onto the queue. Backoff can then exceed the lock duration, jitter is real, and the attempt count lives in state where the workflow definition can govern it.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

The decision that shapes this view

  • Retries are re-enqueued as new scheduled messages rather than abandoned back onto the queue. Abandon-and-redeliver gives no backoff control, no jitter, and an attempt count that lives in the message broker where the workflow cannot govern it.
  • Scheduled enqueue means backoff can exceed the 5-minute lock duration — a 5-minute maximum delay is impossible with abandon — and the attempt count lives in task_execution where it belongs.
  • Full jitter is applied, not equal jitter or none. A thousand tasks failing against the same downstream at the same instant will otherwise retry in a synchronised wave and re-break what is recovering.

Policy and taxonomy

  • Default policy: maxAttempts 5, exponential, initialDelay 2 s, maxDelay 5 m, per task and overridable per task type in the workflow definition.
  • Errors are classified into transient (retry), throttled (retry with the downstream's Retry-After honoured), permanent (dead-letter immediately) and poison (dead-letter, do not re-execute). Classification is handler-declared with a platform default.
  • The circuit breaker is per task type per tenant, with state in Redis, opening after 20 failures in 60 seconds and half-opening with a single probe. A tenant's failing downstream cannot open the circuit for another tenant.

Risks

  • Tasks held by an open circuit are not failing but are also not progressing. They consume no retry budget and no worker, but they do age, so oldest-message-age alerting must distinguish held from stuck.
  • Circuit state in Redis is soft state. If Redis is unavailable the circuit fails closed-as-in-allowed and retries proceed at normal rate, which is the safer failure for a workflow platform but is worth stating.
  • A workflow author can set maxAttempts high enough to hold a task for hours. Validation caps the total retry window at the workflow deadline to prevent this.