Long-Running Agents: Checkpoints, Recovery, and Idempotency

For an agent working for hours, the difficult question is often what has already happened after a restart, timeout, or context compaction. A larger context and a longer loop do not replace task-state records.

Separate task state from conversation

A transcript records discussion. Recoverable state must also identify the objective, constraints, pending actions, confirmed results, failures, remaining budget, and stop conditions. “Done” written by the model is not completion evidence; a readable artifact, tool result, or verification outcome is needed.

A checkpoint captures recoverable state at a particular moment. LangGraph persistence distinguishes per-thread snapshots from cross-thread storage and notes that in-memory checkpoints disappear on restart. A summary helps the next model call understand progress, but should not be the only record of whether an external action occurred.

For a fictional document-organizing task—read sources, draft, write a file, notify the user—state might record the draft version, write-operation identifier, and whether the notification service acknowledged acceptance. Recovery need not infer the last successful step from thousands of transcript lines.

The dangerous gap: executed but not recorded

Suppose the notification service accepts a request, but the client times out before receiving the response. Retrying because no local success record exists can send two notifications. Marking success before sending instead creates the opposite gap: a crash can leave a permanently unsent notification marked complete.

A local checkpoint and an arbitrary remote service do not automatically form one atomic transaction. Temporal's activity execution model separates retryable external work from workflow progress; applications still need defined repeat-execution semantics. A persistence framework alone does not make all external effects exactly once.

Identify logical actions, not attempts

Repeating an idempotent operation does not add another effect. Setting a file to a specific version is easier to retry than appending another paragraph. If an external API supports idempotency keys, every retry of one logical action must reuse its key rather than generate a new UUID.

Associate the key with a task, step, and payload hash. Changing the recipient or content creates a new logical action; reusing the old key could return the old result. The receiver should reject the same key with different parameters, not silently suppress it.

The receiver must insert the key under a unique constraint and make the business change in the same transaction. If it checks whether the key exists and writes the key only at the end, two concurrent requests can both see “no key” and both run. Why the order matters, and how an outbox still handles duplicate delivery, is covered in Relational Data, SQL, and Transactions. Preserve unconfirmed outcomes as unknown rather than treating them automatically as failures.

Recovery is not unconditional replay

Read the last confirmed state and external artifacts before choosing the next step. Reuse stored generation results when appropriate: calling the model again can produce different text and change subsequent actions. Changes to workflow code, tool schemas, or inputs also require checkpoint-compatibility decisions. Temporal workflow execution describes recovery through execution history; separating replayable computation from external I/O is central to that design.

Handle failures by cause. Transient network errors can receive bounded backoff; invalid arguments need repair; missing permissions cannot be solved by retries; long waits need deadlines. How to stop a run together with its subprocesses is covered in bounded agent loops; a long task must also record the side effects completed before cancellation. Stopping future work does not undo past work.

Combine checkpoints with context compaction

Snapshots retain machine-readable progress; summaries supply the explanation the model needs to continue. Update them together where appropriate. What a summary must keep is covered in context engineering; large tool outputs can stay in durable storage while only relevant passages return to context.

Tie completion to the objective. “Draft written and citations checked” differs from “user notified.” Completing one must not overwrite the other with a generic done flag. Budget exhaustion, waiting for external input, and goal achievement likewise need different states.

A useful recovery test interrupts before a write, after the write but before recording it, and after recording. Check artifact correctness, duplicate actions, and whether unknown outcomes can be reconciled. Local files and fake services can test these boundaries without repeatedly sending real messages.

The coding-agent version: git commits as checkpoints

When a coding agent has to finish a project across several sessions, the state record above can live in the repository itself. Anthropic's November 2025 article on harnesses for long-running agents describes one way to do it:

1. Use the first session only to set up the environment. A separate prompt has it write an init.sh that starts the development server, a claude-progress.txt progress log, and the first git commit.
2. Write the requirements as a JSON feature list, every item starting as failing. Later sessions may change only each item's passes field, and the prompt says “It is unacceptable to remove or edit tests.” The article reports that the model damages JSON less often than Markdown.
3. Work on one feature per session. Each session starts by running pwd, reading the progress file and the git log, and running a basic end-to-end check to confirm the previous session did not leave the app broken.
4. Commit and update the progress file before ending. Each commit says what was done, so a bad change can be reverted to a working version with git.
5. Test the way a user would. With only unit tests or curl, the agent often missed features that did not work end to end; asked to use browser automation like a user, it did much better.

Here the git history and the progress file are the checkpoints and the feature list is the completion condition: the same state record described above. A new session picks up with claude -c, codex resume --last, or pi -c; the resume and unattended-run commands for each tool are collected in the table at the end of Harness.

My reading