Long-Running Agents: Checkpoints, Recovery, and Idempotency
For an agent working for hours, the difficult question is often what has already happened after a restart, timeout, or context compaction. A larger context and a longer loop do not replace task-state records.
Agent loops explain observation, action, and verification. This page extends the timescale to recovery across processes and external effects.
Separate task state from conversation
A transcript records discussion. Recoverable state must also identify the objective, constraints, pending actions, confirmed results, failures, remaining budget, and stop conditions. “Done” written by the model is not completion evidence; a readable artifact, tool result, or verification outcome is needed.
A checkpoint captures recoverable state at a particular moment. LangGraph persistence distinguishes per-thread snapshots from cross-thread storage and notes that in-memory checkpoints disappear on restart. A summary helps the next model call understand progress, but should not be the only record of whether an external action occurred.
For a fictional document-organizing task—read sources, draft, write a file, notify the user—state might record the draft version, write-operation identifier, and whether the notification service acknowledged acceptance. Recovery need not infer the last successful step from thousands of transcript lines.
The dangerous gap: executed but not recorded
Suppose the notification service accepts a request, but the client times out before receiving the response. Retrying because no local success record exists can send two notifications. Marking success before sending instead creates the opposite gap: a crash can leave a permanently unsent notification marked complete.
A local checkpoint and an arbitrary remote service do not automatically form one atomic transaction. Temporal's activity execution model separates retryable external work from workflow progress; applications still need defined repeat-execution semantics. A persistence framework alone does not make all external effects exactly once.
Identify logical actions, not attempts
Repeating an idempotent operation does not add another effect. Setting a file to a specific version is easier to retry than appending another paragraph. If an external API supports idempotency keys, every retry of one logical action must reuse its key rather than generate a new UUID.
Associate the key with a task, step, and payload hash. Changing the recipient or content creates a new logical action; reusing the old key could return the old result. The receiver should reject the same key with different parameters, not silently suppress it.
This simplified receiver-side pseudocode shows two operations that require the same transaction:
begin transaction
if operation_key exists:
reject if stored_payload_hash differs
return stored_result
apply the local business change
store operation_key, payload_hash, result
commit transaction
If the business effect occurs in another service, this local transaction is insufficient. The other service needs idempotency support, a queryable business identifier, or a delivery design such as a transactional outbox. Preserve unconfirmed outcomes as unknown rather than treating them automatically as failures.
Recovery is not unconditional replay
Read the last confirmed state and external artifacts before choosing the next step. Reuse stored generation results when appropriate: calling the model again can produce different text and change subsequent actions. Changes to workflow code, tool schemas, or inputs also require checkpoint-compatibility decisions. Temporal workflow execution describes recovery through execution history; separating replayable computation from external I/O is central to that design.
Handle failures by cause. Transient network errors can receive bounded backoff; invalid arguments need repair; missing permissions cannot be solved by retries; long waits need deadlines. Propagate cancellation to active tools and retain completed side effects. Stopping future work does not undo past work.
Combine checkpoints with context compaction
Snapshots retain machine-readable progress; summaries supply the explanation the model needs to continue. Update them together where appropriate. A summary should preserve the goal, accepted decisions, unverified assumptions, and artifact locations. Large tool outputs can stay in durable storage while only relevant passages return to context. See context engineering.
Tie completion to the objective. “Draft written and citations checked” differs from “user notified.” Completing one must not overwrite the other with a generic done flag. Budget exhaustion, waiting for external input, and goal achievement likewise need different states.
A useful recovery test interrupts before a write, after the write but before recording it, and after recording. Check artifact correctness, duplicate actions, and whether unknown outcomes can be reconciled. Local files and fake services can test these boundaries without repeatedly sending real messages.