Error Recovery: Resume Points and Compensating Actions
Long workflows fail. Not because agents are careless, but because the real world is inconsistent. Inputs change. Tools return partial results. Permissions differ across environments. Dependencies stall. Even when every component is “mostly working,” the combined system can hit edge cases that stop progress.
Recovery determines whether agentic systems can be trusted in production. A system that cannot recover becomes expensive and fragile because the only safe response is to restart the workflow from the beginning or route everything to humans. A system that can recover gracefully turns failures into controlled detours.
Competitive Monitor Pick540Hz Esports DisplayCRUA 27-inch 540Hz Gaming Monitor, IPS FHD, FreeSync, HDMI 2.1 + DP 1.4
CRUA 27-inch 540Hz Gaming Monitor, IPS FHD, FreeSync, HDMI 2.1 + DP 1.4
A high-refresh gaming monitor option for competitive setup pages, monitor roundups, and esports-focused display articles.
- 27-inch IPS panel
- 540Hz refresh rate
- 1920 x 1080 resolution
- FreeSync support
- HDMI 2.1 and DP 1.4
Why it stands out
- Standout refresh-rate hook
- Good fit for esports or competitive gear pages
- Adjustable stand and multiple connection options
Things to know
- FHD resolution only
- Very niche compared with broader mainstream display choices
The practical goal is to preserve forward progress without pretending nothing went wrong.
Resume points are the difference between a hiccup and a restart
A resume point is a durable marker that captures enough state to continue the workflow after a failure. Without resume points, failures force the agent to re-plan from scratch and re-run tool calls, which compounds risk and cost.
A good resume point captures:
- The workflow step identifier and the next intended step
- The exact inputs and tool outputs that led to the current state
- The commitments already made, including irreversible actions
- A compact state snapshot that can be replayed deterministically
Resume points should be created at the same boundary moments that drive reliability:
- Before irreversible actions
- After expensive tool calls
- After user approvals
- After successful completion of a major subtask
This turns recovery into a continuation, not a re-creation.
Durable state enables recovery, but it must be scoped
Recovery requires durability, but durability without discipline creates a new failure mode: stale state that survives longer than its validity. The state model must define what is durable and what is ephemeral.
A practical separation:
- Durable workflow state
- Step progress
- Tool receipts and identifiers
- Approved plans and checkpoints
- Durable evidence state
- Retrieved snippets, citations, hashes, diffs
- Audit logs and trace IDs
- Ephemeral working state
- Draft reasoning, tentative hypotheses
- Temporary scratch calculations
This separation keeps recovery honest. The workflow resumes from evidence and receipts, not from whatever story the agent happened to narrate.
Compensating actions replace the fantasy of perfect rollback
Many agent actions cannot be rolled back in the strict sense. You cannot un-send an email. You cannot un-publish a message that was scraped. You cannot guarantee a third-party API undo will restore the original state. Recovery needs a different concept: compensating actions.
A compensating action is a planned response that reduces harm and restores acceptable state after a partial failure.
Examples:
- If a record was created with wrong data, create a corrected record and deprecate the wrong one.
- If a notification was sent prematurely, send a follow-up correction and update the system of record.
- If a file was written incorrectly, write a new version and mark the old one as superseded.
- If a workflow executed in the wrong environment, quarantine outputs and trigger a review.
Compensations are most effective when the system is designed to make them possible: soft deletes, versioned writes, append-only logs, and explicit status fields.
side effects and compensation strategies
| Side effect type | Typical risk | Preferred design | Compensation approach |
|---|---|---|---|
| Database writes | Corrupt state, inconsistent joins | Versioned writes, soft deletes | Write corrected version, deprecate old |
| External messages | Misinformation, trust loss | Draft then send, approvals | Send correction, log incident |
| File generation | Wrong artifact propagated | Immutable artifacts, hashes | Publish corrected artifact, revoke old |
| Payments or credits | Financial harm | Two-step commit, limits | Refund or reverse, escalate to human |
| Access changes | Security exposure | Least privilege, staged rollout | Revoke immediately, audit trail |
| Deployments | Outage, regression | Canary, gates, rollback paths | Roll back, postmortem, patch forward |
A recovery plan should be chosen before incidents happen. That is how reliability becomes a system property instead of a heroic response.
Recovery needs a step model with clear transitions
Recovery is easier when the workflow is a state machine. Every step has:
- Entry conditions
- Expected outputs
- Exit conditions
- Failure modes and the allowed recovery transitions
When the workflow is modeled this way, a failure does not create ambiguity about what can happen next. The system can:
- Retry the step
- Switch to an alternate step
- Pause for approval
- Trigger compensation
- Abort safely with a clear artifact describing what happened
This structure makes agents safer because the model is not inventing the recovery path. The system is enforcing it.
Deterministic replay turns debugging into engineering
When a workflow fails, teams need to know what happened. If the system cannot replay the failure, debugging becomes guesswork, and improvements become slow.
Deterministic replay depends on a few practices:
- Capture tool inputs and outputs with stable identifiers
- Record random seeds or nondeterministic settings when they exist
- Store prompts, policies, and tool versions that influenced decisions
- Keep a compact event log of step transitions
Deterministic replay does not mean the model must always produce identical text. It means the operational steps and tool interactions can be replayed to reproduce the key decisions and outcomes.
Human recovery paths should be designed, not improvised
A large share of recovery in real systems is human-assisted. That does not mean the agent failed. It means the system recognized uncertainty and chose a safer route.
Human recovery works best when the system provides:
- A clear summary of the current state and what failed
- The evidence collected so far
- The proposed next action and the risks
- A simple approval or correction mechanism
- A way to resume the workflow after the decision
When humans are forced to reconstruct context from logs, recovery becomes slow and error-prone. When humans receive structured artifacts, recovery becomes fast and safe.
Recovery is inseparable from observability
Recovery decisions depend on knowing what failed and how often it fails. Observability provides the raw material for recovery improvements:
- Which steps fail most often
- Which tools are the highest sources of retries and timeouts
- Which failure modes lead to compensation
- Which recovery paths succeed
This feedback loop is what turns an agent system into a maturing platform. Without it, each incident is a one-off surprise.
Recovery patterns for distributed side effects
When a workflow touches multiple systems, a strict rollback is rarely available. The reliable approach is to treat the workflow as a series of local steps with explicit receipts, and to coordinate the overall outcome through compensation.
A common pattern is to use a saga-style design:
- Each step commits locally and records a receipt
- If a later step fails, compensations run in reverse order where possible
- If a compensation fails, the workflow enters a quarantine state that requires human resolution
This approach is not theoretical. It matches how real systems behave: partial success is normal, and reliability comes from making partial success safe.
The outbox idea prevents lost actions
A subtle recovery bug happens when a system updates state and then fails before it can publish the corresponding action or notification. The state says the work happened, but downstream systems never hear about it.
An outbox-style approach avoids this:
- Record the intended side effect as a durable event alongside the state update
- Process the event asynchronously with retries and deduplication
- Mark the event as delivered only when the side effect receipt is stored
This turns recovery into a deterministic process. If the worker crashes, the outbox still contains the work to be delivered. If the delivery is duplicated, idempotency keys prevent double execution.
Compensations should be tested like features
Compensation logic is often written in a rush after an incident. That creates fragile recovery paths that fail under pressure. A better approach treats compensation as a first-class feature:
- Unit tests for compensation transitions
- Integration tests against staging systems
- Fault-injection tests that trigger mid-workflow failures
- Runbooks that specify who owns each quarantine state
The goal is not to eliminate failures. The goal is to ensure failures land in known, supportable states.
Orchestration engines make recovery operational
Recovery becomes easier when workflows run inside an orchestration engine that already understands retries, timeouts, durable state, and step transitions. Even lightweight orchestration adds real value:
- Durable step state and progress markers
- Centralized retry budgets and backoff policies
- Built-in cancellation and timeouts
- Visibility into where workflows are stuck
When orchestration is absent, recovery logic tends to spread across ad hoc prompts and tool calls. That makes incidents harder because there is no single place to see the real workflow state.
Governance and audit are recovery tools
Recovery is not only technical. It is also governance. When a system can show what happened, who approved what, and which compensations were applied, incident response becomes calmer and faster.
Useful governance artifacts:
- Approval receipts for high-stakes actions
- Audit logs for tool invocations and external side effects
- Version stamps for prompts, policies, and tool configurations
- Post-incident summaries tied to trace identifiers
These artifacts are what allow organizations to trust automation. Without them, every failure becomes a reputational risk.
Keep exploring on AI-RNG
- Agents and Orchestration Overview: Agents and Orchestration Overview
- Nearby topics in this pillar
- Tool Error Handling: Retries, Fallbacks, Timeouts
- State Management and Serialization of Agent Context
- Deterministic Modes for Critical Workflows
- Human-in-the-Loop Checkpoints and Approvals
- Cross-category connections
- User Reporting Workflows and Triage
- Blameless Postmortems for AI Incidents: From Symptoms to Systemic Fixes
- Series and navigation
- Deployment Playbooks
- Capability Reports
- AI Topics Index
- Glossary
More Study Resources
- Category hub
- Agents and Orchestration Overview
- Related
- Tool Error Handling: Retries, Fallbacks, Timeouts
- State Management and Serialization of Agent Context
- Deterministic Modes for Critical Workflows
- Human-in-the-Loop Checkpoints and Approvals
- User Reporting Workflows and Triage
- Blameless Postmortems for AI Incidents: From Symptoms to Systemic Fixes
- Deployment Playbooks
- Capability Reports
- AI Topics Index
- Glossary
Books by Drew Higgins
Christian Living / Encouragement
God’s Promises in the Bible for Difficult Times
A Scripture-based reminder of God’s promises for believers walking through hardship and uncertainty.
Bible Study / Spiritual Warfare
Ephesians 6 Field Guide: Spiritual Warfare and the Full Armor of God
Spiritual warfare is real—but it was never meant to turn your life into panic, obsession, or…
