A workflow executes over days or weeks while individual workers restart many times. How should workflow state, timers, retries, heartbeats and task ownership be designed?

The requirement

The workflow's progress must outlive every process that executes it. A worker restarting, being deployed, crashing or being rescheduled must not lose position, and a workflow paused for three days waiting on a human must resume correctly on a machine that did not exist when it started.

The components

• Durable workflow state, external to the worker. Each completed step is persisted, so recovery resumes from the last completed step rather than from the beginning. This is the core property and everything else supports it.
• Durable timers. "Wait 30 days then escalate" cannot be a sleep in a process. It must be a persisted timer that fires regardless of what happened to the process that scheduled it.
• Task ownership with leases and heartbeats. A long-running step must periodically confirm it is alive, so a dead worker's task is reassigned — and the heartbeat must confirm progress, not merely liveness, or a stuck worker holds its task indefinitely.
• Declared retry policy per step, since steps differ: a transient API call retries aggressively, a human-approval step does not retry at all.
• Idempotent steps, because reassignment after a lease expiry means a step can execute twice — the worker may have completed the work and died before recording it.
• A terminal state for every path, including compensation failure, which must escalate to a human rather than retry forever.

What a workflow engine changes

Using an engine such as Temporal moves this machinery out of application code. Its value is not that it prevents failure — it makes the recovery code someone else's problem, and hand-written orchestration of long-running processes is where application-level implementations fail most reliably.

The costs are real: a new operational dependency, a programming model with genuine constraints (determinism in workflow code, versioning of workflows that are already running), and a new place for outages.

The versioning problem that surprises teams

A workflow running for three weeks may need to survive a deployment that changes its own definition. New executions should use the new logic; in-flight ones must either continue with the old or be migrated deliberately. This is a first-class concern in long-running workflows and it has no easy answer — which is itself an argument for using an engine that has thought about it rather than inventing a scheme.

The scoping judgement

Not everything needs durable execution. It earns its cost when workflows are long-running, numerous, involve external parties, and are expensive to get wrong — an onboarding that spans days, a payout with compliance checks, a claims process. A three-step synchronous flow does not need it, and adopting it universally imposes a real learning and operational cost on work that never had the problem.