Agentic AI Engineering · Module 09 of 16

Reliability Engineering

Agents fail in more ways than 'the model was wrong'. Networks time out, tools return errors, retries duplicate actions, and loops consume budgets. Reliability engineering makes those failures bounded and recoverable.

Engineering focus: Budgets, retries, timeouts, idempotency and failure classification

In plain English

What problem are we solving?

Framework names come later. First understand what the application is doing and where its control boundaries belong.

Running case · Operations Investigator

Across the course, imagine an assistant that investigates operational problems using narrow tools. The design progressively adds plans, specialist agents, approvals, reliability controls and MCP while keeping model output separate from execution authority.

Learning objectives

By the end of this module, you should be able to:

Classify transient, permanent and policy failures before choosing a recovery action.
Use bounded retries, deadlines and budgets instead of retrying forever.
Use idempotency or deduplication to prevent duplicate side effects.

Not every failure is retryable

A timeout or temporary rate limit may succeed later. Invalid arguments, authorization denial and policy rejection usually will not. Retrying a permanent failure wastes budget and can amplify incidents.

Retries must be bounded

Define maximum attempts, backoff and a deadline. The retry loop belongs to application policy, not to an unconstrained model instruction such as 'keep trying until it works'.

Side effects need idempotency or deduplication

If a network response is lost after a successful restart, a retry can perform the same action twice. An idempotency key or operation ledger lets the tool recognize that the requested side effect already happened.

Budgets create a safe stopping point

Bound model turns, tool calls, wall-clock time and cost. When the budget is exhausted, return the evidence gathered so far and escalate rather than silently continuing.

Worked examples

Failure decision

timeout / 429        → bounded retry
invalid arguments     → fix request, do not blind-retry
permission denied     → stop / escalate
policy rejected       → stop
unknown partial write → verify before retry

Recovery depends on failure class.
Verification is essential when the side effect may already have occurred.

Idempotent action

restart_service(
  host='web01',
  service='nginx',
  idempotency_key='incident-123-step-4'
)

Repeated delivery of the same request can be detected.
The key represents one intended operation, not every retry as a new action.

Try it · core lab · account-free

Recover from transient failure without duplicating side effects

Goal: Retry a flaky read within a fixed budget and prove that a repeated side-effect request executes only once.

Before you start

Python 3 only. Failures and side effects are simulated locally.

STEP 1

Run the reliability lab

python3 public/labs/agentic-ai/09_reliability_engineering.py

STEP 2

Watch bounded retry

A transient read fails twice and succeeds on the final permitted attempt.

STEP 3

Replay the same change request

The operation ledger recognizes the idempotency key and prevents a duplicate simulated restart.

Download this Python lab Download all current labs

Verify

The transient operation never exceeds its maximum-attempt budget.
The side-effect counter remains one after a replay with the same idempotency key.

Expected outcome

Transient and permanent failures take different paths.
A retry cannot silently multiply a completed side effect.

Break / fix · diagnose

If attempts exceed the limit, inspect the loop condition and increment location.
If the side-effect counter becomes two, verify that the ledger is checked before execution.

Production view

Reliable agent runtimes treat model calls and tools like distributed-system dependencies: every call needs deadlines, retry policy, correlation, idempotency where possible, and an explicit escalation path.

Avoid these traps

Common mistakes

Retrying every error with the same policy.
Letting the model decide to retry indefinitely.
Retrying a side effect without first checking whether it already succeeded.
Having no overall turn, time or cost budget.

Knowledge check

Explain the control decision, not just the vocabulary.

Open every answer only after committing to your own answer. Opening all four answers records knowledge-check activity in My Learning.

1Which failure is most likely to be retryable?

A transient timeout or rate limit — provided the retry is bounded and the operation is safe to repeat.

2Should an authorization denial be blindly retried?

No — it is normally a permanent/policy failure until permissions or context change.

3Why use an idempotency key for a side-effecting tool?

So delivery retries can represent the same intended operation without executing the side effect multiple times.

4What should happen when the agent exhausts its run budget?

Stop safely, return available evidence/state and escalate or ask for intervention rather than continuing indefinitely.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.