Agentic AI Engineering · Module 14 of 16

Failure Recovery

A multi-step agent can fail after doing some work. Recovery means knowing what already happened, what can be retried, what must be undone or compensated for, and when to stop and ask a human.

Engineering focus: Checkpointing, compensation, resume and safe escalation

In plain English

What problem are we solving?

A multi-step agent can fail after doing some work. Recovery means knowing what already happened, what can be retried, what must be undone or compensated for, and when to stop and ask a human.

Framework names come later. First understand what the application is doing and where its control boundaries belong.

Running case · Operations Investigator

Across the course, imagine an assistant that investigates operational problems using narrow tools. The design progressively adds plans, specialist agents, approvals, reliability controls and MCP while keeping model output separate from execution authority.

Learning objectives

By the end of this module, you should be able to:

Differentiate retry, resume, rollback/compensation and escalation.
Persist enough execution state to avoid repeating completed side effects.
Design a recovery path for partial completion and verification failure.

Recovery begins with durable state

For important workflows, persist step status, operation IDs, tool results and verification evidence outside the model context. If the process restarts, the runtime can reconstruct what completed instead of guessing from conversation text.

Retry is not the same as resume

Retry repeats a failed operation under a bounded policy. Resume continues from a known checkpoint. Replaying the whole plan can duplicate already-completed side effects.

Some actions need compensation, not rollback

Distributed systems often cannot truly undo an action. Instead define a compensating action: for example, if a deployment changed traffic before a later step failed, restore the prior route or deploy the known-good version. Compensation must itself be authorized and verified.

Escalate when state is uncertain

If the runtime cannot determine whether a consequential action succeeded, do not blindly repeat it. Reconcile current state using an idempotency key or read-after-write verification, then escalate if certainty cannot be restored.

Worked examples

Checkpointed run

01 inspect         COMPLETE
02 propose         COMPLETE
03 approve         COMPLETE
04 restart nginx   COMPLETE op=OP-77
05 verify          FAILED timeout

resume → verify current service state
not → repeat restart blindly

The checkpoint protects against duplicate side effects.
Recovery starts by reconciling the real environment.

Compensation pattern

change traffic → succeeds
deploy new version → fails
compensation → restore previous traffic target
verify → healthy
escalate with evidence

Compensation is an explicit new action.
It should be policy-controlled just like the original change.

Try it · core lab · account-free

Recover a partially completed agent run

Goal: Resume from a checkpoint, avoid duplicate side effects and execute a compensating action when verification fails.

Before you start

Python 3 only.

STEP 1

Run the recovery simulation

python3 public/labs/agentic-ai/14_failure_recovery.py

STEP 2

Observe the checkpoint

A side effect is marked complete before a later verification failure.

STEP 3

Resume safely

The runtime reads state, skips the completed side effect and performs reconciliation/compensation.

Download this Python lab Download all current labs

Verify

The side effect counter remains one after resume.
The compensation path runs only after the verification failure.
The final state includes a clear escalation/recovery record.

Expected outcome

You can explain why restarting the entire agent loop is unsafe after partial execution.
You can distinguish retry, resume and compensation.

Break / fix · diagnose

If the side effect executes twice, persist/check operation completion before dispatch.
If recovery guesses current state, add an explicit read/reconciliation step.

Production view

Long-running agent systems need workflow-style durability: checkpoints, idempotency, reconciliation, compensating actions and escalation boundaries. Conversation history alone is not an execution journal.

Avoid these traps

Common mistakes

Retrying the entire plan after any exception.
Assuming every external side effect can be rolled back.
Keeping execution state only in model context.
Repeating an uncertain action before reconciling actual state.

Knowledge check

Explain the control decision, not just the vocabulary.

Open every answer only after committing to your own answer. Opening all four answers records knowledge-check activity in My Learning.

1What is the difference between retry and resume?

Retry repeats a failed operation; resume continues from a known checkpoint while preserving completed work.

2When is a compensating action useful?

When the original side effect cannot truly be rolled back but another controlled action can restore an acceptable state.

3What should happen if you cannot tell whether a payment/restart/change already succeeded?

Reconcile using durable operation identity and current-state checks before retrying; escalate if certainty cannot be restored.

4Why is conversation history insufficient as an execution journal?

It is not a durable, authoritative record of side effects, operation IDs, verification state and transaction boundaries.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.