AI Agent Orchestration Platform  ·  View 19 of 32  ·  4 · Runtime

Durable Execution and Replay

What survives a crash, how failure is classified, and where a run goes when recovery is exhausted.

Editable source SVG draw.io All views
Progress Step Completes worker result Checkpoint Write before ack Queue Ack lock released Failure Process Crash pod evicted Node or Zone Loss AZ outage Provider Outage model 5xx Detect Lock Expiry no renewal Heartbeat Miss orchestrator Classify Failure eight classes Recover Redeliver Message same session Idempotency Guard step key seen Scheduled Retry backoff + jitter Provider Failover next route Resume Load Checkpoint last committed Pinned Definition original hash Continue Step no re-execution Exhausted Dead Letter after max attempts Compensate reverse effects Human Escalation with full trace transient provider permanent run continues on partial reversal Durable Execution — Checkpoint, Failure, Recovery and Replay Application we own Data store Queue / topic Risk / gap Interface / broker Security / platform Decision point Person or role synchronous failure / alternate The checkpoint is written before the queue acknowledgement, so a crash duplicates work rather than losing it. v 1.0 · owner Runtime Engineering · date 2026-08

The ordering that matters

  • The checkpoint is written before the queue acknowledgement; a crash between them duplicates work rather than losing it
  • Duplicate work is made safe by the idempotency guard at the step key, which is why at-least-once is an acceptable contract
  • Resume loads the last committed checkpoint and the originally pinned definition, so a mid-run recovery cannot change behaviour

Failure classes

  • Transient, provider, tool, agent or model, policy, authentication, user action and permanent workflow error
  • Each class has its own retry policy; only transient and provider classes retry automatically
  • Policy and authentication failures fail closed and escalate — retrying an authorisation denial is a defect, not resilience

Risks

  • Compensation is best-effort against systems that may not support reversal; irreversible actions are the reason human approval exists
  • Replay of a run whose workflow version has been retired requires the retired version to remain resolvable — retirement drains rather than deletes
◀  Multi-Agent Coordination All views Human-in-the-Loop  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-08 · 32 views · Microsoft Azure19-durable-execution-and-replay