Durable Execution

A long-running business process — payment, verification, waiting three days, notification, fulfilment — must survive worker crashes, deploys and infrastructure failures mid-flow. Building that by hand means creating a persisted state machine, idempotent steps, a restart-surviving timer service, compensation in reverse order, per-step retry policies, visibility into stuck instances, and a versioning story for in-flight work.

That list is a workflow engine, and teams arrive at one incrementally, under-tested in exactly the failure paths that justify its existence.

Durable execution provides it directly. The workflow is written as ordinary sequential code, and the engine persists the result of every step. After a crash, it replays the recorded history to reconstruct local state and continues from the first unexecuted step.

Why it matters

The value is not that failures are handled — it is that the application code does not contain the handling. The business process reads as the business process, and the durability is a property of the runtime rather than a concern distributed through the code.

The operational benefit is equally significant: every workflow instance is queryable. "Where is customer 12345's onboarding and why is it stuck" has an answer, which in a hand-rolled or choreographed implementation requires reconstructing the flow from logs across several services.

Implementation patterns

• Deterministic workflow code, since replay must reproduce the same decisions. No now(), no random values, no direct network or database calls, no iteration over unordered collections. All non-determinism moves into activities, whose results are recorded and replayed rather than re-executed.
• Explicit version gating for in-flight workflows. An instance started three weeks ago replays against today's code, so any change to the step sequence breaks replay unless old instances take the old path. This must be planned before the first long-running workflow ships, because retrofitting is painful.
• Idempotent activities, because an activity may be re-executed after a failure between execution and recording.
• Timers and waits as first-class workflow constructs, which survive restarts by construction.
• Compensation written as ordinary code in the workflow's error path, rather than as a separate compensation framework.
• History size managed — long-lived workflows accumulate history, and continue-as-new or similar mechanisms are needed to bound it.
• The engine treated as production infrastructure, with its own scaling, storage growth and failure modes.

Industry example

Temporal is the most widely deployed implementation, having grown out of the Cadence work at Uber, where the problem was exactly this: long-running processes spanning many services — trip lifecycle, payments, driver onboarding — that had to survive continuous deployment and infrastructure churn.

Its adoption pattern is consistent: organisations reach it after building two or three hand-rolled sagas and discovering that the state machine, the timers, the retries and the visibility are the same code each time, and that the in-flight versioning problem has no good answer when built ad hoc.

Failure scenarios

• Non-determinism in workflow code — a map iteration order, a missed Date.now(). Almost always the first production bug, and it manifests as a replay failure on an instance that was working.
• No versioning strategy, so a deploy breaks every in-flight workflow.
• Non-idempotent activities, producing duplicate side effects on retry.
• Business logic in activities that should be in the workflow, and vice versa, blurring what is replayed.
• Unbounded history on long-lived workflows.
• Using it for a three-step synchronous operation in one service, where a transaction and an outbox are simpler with no determinism constraint.
• Treating the engine as free infrastructure, without capacity planning or an owner.
• Workflows that call each other synchronously in deep chains, recreating the coupling the pattern was meant to make visible.

Trade-offs

The determinism constraint is a genuine change in how code is written, and it is routinely underestimated. Every developer touching workflow code must understand it, and the failure mode — a replay error on an instance that ran fine yesterday — is confusing until the model is internalised.

The engine is also an operational dependency with real weight: a database, a service to run, scaling characteristics, upgrade cycles, and an outage that stops every workflow in the organisation.

The trade is a determinism constraint and a significant operational dependency in exchange for correctness in the failure paths and complete visibility into in-flight work. For workflows measured in hours to months, spanning services, where being stuck invisibly is expensive, it is decisively worth it. For short synchronous operations it is heavy machinery, and the honest summary is that you are choosing between an engine somebody else has debugged and one you will debug yourself, in production, in the failure paths.

Interview question

"Our onboarding flow spans five services and takes up to two weeks, and we lose instances every time we deploy. Tell me how durable execution fixes that, then tell me what happens to a workflow started last Tuesday when we change the order of two steps this morning."