Durable Agent Execution and Recovery

An agent thirty minutes into a task, having made eleven tool calls, four of which wrote to external systems, hits a 529 from the model API. What happens next is an architecture question, and most agent frameworks answer it badly by default: the process dies, the conversation is gone, and the four writes have already happened.

This is not an AI problem. It is the problem distributed systems solved with workflow engines, and agents inherit it in a sharper form because their steps are expensive, non-deterministic and externally visible.

The state that has to survive

An agent loop's state is larger than its message list. Durable execution means persisting, at minimum:

• the message history, including tool calls and their results
• the step index or cursor, so resume knows where it stopped
• external effects already committed, so they are not repeated
• the plan or task list, if the agent maintains one separately from the conversation

Checkpointing after every step is the simple policy and usually the right one, because agent steps are seconds-to-minutes long and a checkpoint write is milliseconds. The interesting question is not how often to checkpoint but what a checkpoint has to make true.

Idempotency is the hard half

Resume is only safe if replaying a step is safe. The model's own output is not the problem; sampling is non-deterministic but a fresh generation is usually acceptable. The problem is tool calls with side effects. Resuming after a crash that happened between "issued the payment" and "recorded that we issued the payment" must not issue a second payment.

The standard fix applies unchanged: give every effectful call a client-generated idempotency key derived from the run ID and step index, and have the tool layer deduplicate on it. This pushes a requirement onto tool authors that direct API wrappers rarely satisfy, and it is the single most common gap in agent systems that otherwise checkpoint correctly.

A weaker but useful discipline is to separate reads from writes in the loop, batch the writes, and place them as late as possible, so the window in which a crash can leave inconsistent external state is small and well defined.

Human-in-the-loop is the same machinery

An agent that pauses for approval before a destructive action is a durable execution problem wearing a different hat. The run must suspend, persist, survive the process that hosted it, and resume on an event that arrives minutes or days later from a different request. Systems that implement checkpointed resume get approval gates almost for free; systems that keep agent state in process memory cannot implement approval gates at all without inventing the same persistence layer under a different name.

When it breaks

Checkpoint size grows with the window. Persisting the full message history every step means writing a 150,000-token blob repeatedly. Store deltas, or store a pointer to immutable message records and checkpoint only the cursor.

Resume after a model version change is not a resume. The persisted history was generated by one model; replaying it into another can produce a trajectory that contradicts its own earlier steps. Pin the model per run, or treat a version change as a reason to restart rather than resume.

Compaction and checkpointing interact badly. If the checkpoint stores the compacted window rather than the raw history, a bad compaction becomes permanent and unrecoverable. Persist raw, compact for the model.

Poison-pill loops. Automatic resume on failure will faithfully retry the step that crashed the process, forever. Retries need a per-step attempt counter that escalates to a human, which is the one part of this that cannot be borrowed unchanged from ordinary workflow engines, because the agent can also decide to retry and neither layer knows about the other's counter.