Durable Execution

A business process running over days or weeks will outlive many process lifetimes. Workers are deployed, rescheduled, crash and restart. The workflow's progress must survive all of it, and a process that holds its state in memory or in a sleep call cannot provide that.

Durable execution persists each completed step and each pending timer externally, so recovery resumes from the last completed step rather than from the beginning.

Why it matters

Hand-written orchestration of long-running processes is where application-level implementations fail most reliably. The failure is not dramatic — it is a workflow that silently stopped three weeks ago because the worker holding it was replaced, discovered when a customer asks why nothing happened.

Implementation patterns

• Durable state external to the worker, recorded per completed step.
• Durable timers. "Wait thirty days then escalate" cannot be a sleep in a process; it must be a persisted timer that fires regardless of what happened to the scheduler.
• Task ownership with leases and heartbeats, so a dead worker's task is reassigned — and the heartbeat must confirm progress, not merely liveness, or a stuck worker holds its task indefinitely.
• A retry policy declared per step, since a transient API call and a human-approval step have nothing in common.
• Idempotent steps, because lease expiry and reassignment mean a step can run twice: the worker may have completed the work and died before recording it.
• A terminal state for every path, including compensation failure, which must escalate to a human rather than retry forever.
• A workflow versioning strategy, because a workflow running for three weeks may need to survive a deployment that changes its own definition — new executions on new logic, in-flight ones continued on the old or migrated deliberately.

Industry example

Workflow engines such as Temporal exist to move this machinery out of application code. The value is not that failure becomes impossible — it makes the recovery code someone else's problem, which for long, numerous, externally-dependent workflows is a large saving.

The costs are real and should be stated: a new operational dependency, a programming model with genuine constraints such as determinism in workflow code, and a new place for outages.

Failure scenarios

• Workflow state in process memory, lost on every deployment.
• Sleep-based timers, which do not survive a restart.
• Heartbeats that confirm liveness rather than progress, so a stuck worker is never reclaimed.
• Non-idempotent steps, duplicating effects on reassignment.
• No versioning plan, so a deployment breaks every in-flight workflow.
• Compensation retried forever on a permanent failure, with no escalation.

Trade-offs

Durable execution imposes a programming model with real constraints, a new system to operate, and a learning cost on every engineer. Applied universally it taxes work that never had the problem — a three-step synchronous flow does not need it.

It earns its cost when workflows are long-running, numerous, involve external parties, and are expensive to get wrong: onboarding spanning days, payouts with compliance steps, claims processes, provisioning sequences. The scoping decision is the important one, and adopting it as a default for all asynchronous work is the common overreach.

Interview question

"A workflow has been waiting fourteen days for a customer to upload a document. In that time you have deployed forty times and replaced every worker. What in your design guarantees it still fires, and what happens if the workflow's own code changed during that period?"