Durable Agent Execution

Treat an agent the way Kubernetes treats a workload: failure is normal, so recovery is part of the design. Model inference can time out, return a different answer on retry, or fail midway through a streamed response. Tools and downstream APIs can throttle, disconnect, or accept a write without returning confirmation. Workers restart. Deployments replace processes.

None of those events should erase hours of work or force the user to restart the interaction. Durability is not an advanced feature for autonomous swarms. It is a core property of any production agent that performs expensive work, calls unreliable systems, or waits for people.

The agent is a workflow, not a process

An agent may look like one continuous conversation, but its execution should be modeled as a sequence of durable steps. The process running a turn is disposable. The workflow state is not.

Durability matters even when nothing fails. Useful agent work is often asynchronous. Research can run for hours. A tool may wait on an external job. An approval may arrive tomorrow. The agent should consume compute while it is working, persist what happened, and suspend while it has nothing to do.

stateDiagram-v2
    [*] --> Running: message or trigger
    Running --> Suspended: turn complete
    Running --> Waiting: approval, timer, or external job
    Waiting --> Running: authorised event
    Suspended --> Running: next message
    Running --> Recovering: worker, model, or tool failure
    Recovering --> Running: resume from safe boundary
    Running --> Completed: verified outcome
    Completed --> [*]

Suspension is not completion. A suspended session keeps its identity, history, pending obligations, and authorization requirements. It releases worker and model resources until a valid event resumes it.

Persist a replayable history

Checkpoint at every turn and consequential side-effect boundary. The durable record should include:

• the ordered event history and current workflow state
• the active prompt, model, tool, workflow, and policy versions
• completed model and tool results, including idempotency keys
• pending timers, jobs, approvals, and their expiry rules
• the conversation, artifact, actor, tenant, and channel identifiers
• the next valid transitions from the current state

This history is more than a saved transcript. It is the evidence needed to decide what has already happened, what may happen next, and whether replay is safe.

Replay orchestration, not side effects

Recovery must replay deterministic orchestration decisions without repeating completed nondeterministic work. Never call the model again merely to reconstruct what it previously decided. The second answer may differ. Never repeat a write tool unless its idempotency contract makes the retry safe.

Keep deterministic workflow logic outside nondeterministic activities:

• Workflow logic decides sequence, waits, branches, retries, and completion conditions. It must replay consistently.
• Activities perform model inference, tool calls, network I/O, and other nondeterministic work. Their results must be recorded.

A checkpoint alone is not durable execution. The runtime also needs failure detection, heartbeats for long activities, leases or locks to prevent duplicate recovery, retry policy, workflow version guards, and a way to resume from the last safe boundary. These mechanics are part of The Harness for a single agent and the Orchestrator when execution spans a wider pipeline.

Make side effects retry-safe

Distributed systems cannot always tell the difference between a failed operation and a lost acknowledgement. A payment, issue creation, email, or database write may have succeeded even when the caller received a timeout.

Every consequential tool needs an explicit retry contract:

• use a stable idempotency key for the logical action
• record the tool request before dispatch and its result after completion
• query downstream state before retrying an ambiguous write
• define compensation when an action cannot be made idempotent
• separate safe reads from writes that require stronger controls
• deduplicate repeated messages, webhooks, and approval events

Exactly-once execution is rarely available end to end. Design for at-least-once delivery with idempotent effects and auditable recovery.

Wait without keeping a model alive

Long-running does not mean one model call, one worker, or one context window stays open for days. The durable workflow should stop at explicit wait states and resume on a new event:

• a user sends the next message
• an authorized person approves or rejects an action
• an external job completes
• a timer or deadline fires
• a recovery trigger detects a stalled activity

The resumed turn should reconstruct only the context it needs from durable state. This is the execution counterpart to Runtime Context Management: preserve task state outside the model, then use a fresh, healthy context for the next step.

Resume against the current user and state

A shared chat thread may contain several people. The person who started a task, supplied context, and approved an action may be different actors. A durable session must not freeze authorization at the beginning or assume any reply in the thread is a valid approval.

Bind every resumable event to an authenticated actor, channel, session, and expected state version. Before continuing, verify:

1. Identity: who sent the message or approval?
2. Authority: may that actor approve this action now?
3. Freshness: does the event reference the current proposal and state version?
4. Ordering: were earlier events applied, and is this event a duplicate?
5. Scope: which tenant, thread, artifact, credentials, and policy apply?

Reject or request a new approval when the proposal, data, permissions, or policy changed during suspension. Resolve user-scoped credentials at execution time, not when the session first paused.

Technologies implement the pattern

Durability is an architectural requirement, not a property owned by one framework. Vercel eve is one current implementation. It models conversations as checkpointed durable workflows and can suspend approval-gated actions without holding compute.

Other workflow engines and agent runtimes may implement the same responsibility differently. Evaluate them against the required semantics: replay safety, side-effect handling, suspension, recovery, workflow versioning, identity, authorization, and exportable history.

Related chapters

• The Agent Loop: the inner think-act-observe cycle that durable execution wraps.
• The Harness: owns the single-agent runtime and its execution boundaries.
• Runtime Context Management: keeps model context healthy across resumed turns.
• Controls & Autonomy: defines approvals and authority for consequential actions.
• Execution Ledger: stores the durable, queryable execution record.
• Orchestration Triggers: supplies recovery, timer, and external resume events.
• Vercel eve: one technology implementation of durable agent sessions.