LLMOps & AI Platform Engineering · Module 12 of 16

Operating RAG & Agents

RAG and agents add more moving parts around the model. Good operators trace retrieval, context, tool calls, approvals and final answers separately so failures can be localized and recovered safely.

Engineering focus: Operating RAG pipelines and agent runtimes

In plain English

What problem are we solving?

RAG and agents add more moving parts around the model. Good operators trace retrieval, context, tool calls, approvals and final answers separately so failures can be localized and recovered safely.

Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.

Animated mental model

Follow a grounded agent request through retrieval, reasoning and tools

Each boundary has different evidence, risks and recovery actions.

Speed

● Loading interaction engine…

● PAUSEDStart one run

01User requestrun_idNOW

02Retrievalfresh + relevantNOW

03Modelreason / composeNOW

04Toolauthorized actionNOW

05Approvalexact actionNOW

06Verified resultevidence + answerNOW

STEP 1/6Start one run

Assign a run identity before retrieval, model and tool stages so evidence remains correlated.

run=RUN-42

Learning objectives

By the end of this module, you should be able to:

Separate retrieval quality, model reasoning and tool execution as different operational boundaries.
Recognize stale indexes, irrelevant retrieval, tool failure and approval denial from evidence.
Define budgets for agent turns, tool calls, time and cost.
Use run IDs, checkpoints and safe escalation for long or partially completed workflows.

🎓 Learning Coach follows this block when it is centered in your browser

RAG has an operating lifecycle beyond vector search

RAG quality depends on ingestion freshness, chunking, index health, query formulation, retrieval/reranking and how context is assembled for the model.

Track source/version identifiers so an answer can be traced back to the exact retrieved material.

Ingestion freshness and failed documents.
Retrieval hit quality and empty/irrelevant results.
Context size and truncation.
Citation/provenance availability.

Common beginner misconception

A fluent answer is not evidence that retrieval was correct.

🎓 Learning Coach follows this block when it is centered in your browser

Agent operations need bounded loops and visible tool decisions

An agent may plan, call tools, inspect results and repeat. Operations must record each turn and tool boundary while enforcing maximum turns, time and cost.

The model proposes actions; runtime policy decides which actions are permitted and whether approval is required.

Run/turn/tool-call identifiers.
Tool authorization independent of model intent.
Budgets for turns, time, tokens and spend.
Stop/escalate behavior when progress stalls.

Common beginner misconception

Prompt instructions are not an authorization system.

🎓 Learning Coach follows this block when it is centered in your browser

Human approval is a runtime state transition, not a chat message

High-impact tool calls should stop at an approval checkpoint containing the exact proposed action, target and arguments. The system should revalidate state before execution because reality may change while waiting.

Approval events belong in the same run trace so operators can distinguish model delay, human delay and execution delay.

Bind approval to exact tool/target/arguments.
Record approver identity and expiry.
Revalidate policy and target state before execution.
Resume from a checkpoint rather than replaying the entire run blindly.

Common beginner misconception

“Approved earlier” does not mean “approved forever.”

🎓 Learning Coach follows this block when it is centered in your browser

Recover from partial failure using checkpoints and evidence

Long-running workflows can fail after some steps succeeded. Before retrying, determine what definitely happened, what definitely did not, and what is uncertain.

Checkpoint state, idempotency and compensating actions prevent blind re-execution of side effects.

Checkpoint after meaningful state transitions.
Use idempotency keys for repeatable side effects.
Verify uncertain outcomes before retry.
Escalate ambiguous/high-risk state to a human.

Common beginner misconception

Retry is not recovery when the previous outcome is unknown.

Interactive simulator

RAG + agent incident simulator

Inject retrieval, tool and approval failures, then see which signal localizes the problem.

TEACHING MODEL

● Loading simulator engine…

Retrieval—

Tool—

Approval—

Diagnosis—

This simulator teaches relationships and diagnostic reasoning. Its synthetic values are not vendor benchmarks, hardware-sizing evidence or production SLO measurements.

Worked examples

Localize a bad grounded answer

retrieval_score=0.21
source_version=policy-v3 (current=v5)
model_followed_context=true

Diagnosis: retrieval/index freshness

The model may be behaving exactly as instructed.
Fix the stale retrieval source before changing the model.

Bound an agent run

max_turns=8
max_tool_calls=12
deadline=45s
cost_budget=$0.20
approval_required=[restart_service]

Budgets turn runaway behavior into a controlled terminal state.
Authorization remains outside the model.

Try it · core lab · account-free

Diagnose RAG and agent failures from run evidence

Goal: Use synthetic run traces to distinguish retrieval, tool, approval and partial-execution failures.

Before you start

Python 3 only. No vector database or model is required.

STEP 1

Diagnose stale retrieval

python3 public/labs/llmops/12_rag_agent_operations.py stale-index

STEP 2

Diagnose tool failure

python3 public/labs/llmops/12_rag_agent_operations.py tool-error

STEP 3

Inspect approval wait

python3 public/labs/llmops/12_rag_agent_operations.py approval

STEP 4

Handle uncertain side effect

python3 public/labs/llmops/12_rag_agent_operations.py uncertain

Download this Python lab Download all current labs

Verify

stale-index reports diagnosis=retrieval_freshness.
tool-error reports diagnosis=tool_execution_failure.
approval reports diagnosis=human_approval_wait.
uncertain reports action=verify_before_retry.

Expected outcome

You can localize failure to retrieval/model/tool/approval boundaries.
You can explain why checkpoint and idempotency evidence matter.

Break / fix · diagnose

If every failure is classified as model error, inspect the stage-level evidence.
If uncertain outcomes immediately retry, add verification before side effects repeat.

Production view

RAG and agent platforms need stage-level traces, source provenance, bounded loops and resumable checkpoints so incidents are diagnosable and actions are safe.

Production Engineer checklist

Before you call this concept production-ready:

RAG ingestion freshness and failures are measurable. Retrieved sources carry provenance/version identifiers. Agent turns/tool calls/time/cost are bounded. Tool authorization and approvals are runtime-enforced. Run IDs correlate retrieval, model, tool and approval stages. Checkpoints/idempotency support safe recovery from partial failure.

This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.

Avoid these traps

Common mistakes

Blaming the model for stale or irrelevant retrieval.
Allowing unlimited agent turns/tool calls.
Treating approval as free-form chat instead of a bound runtime event.
Retrying a timed-out side effect without verifying whether it already happened.

Knowledge check

Reason about the system, not just the vocabulary.

Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.

1What is the first place to look when citations point to an old document version?

Ingestion/index freshness and source-version provenance.

2Why should agent tool authorization be outside the model prompt?

Authorization must be deterministic and enforceable even when the model is manipulated or mistaken.

3What should an approval record bind to?

The exact tool/action, target, arguments, approver and validity/expiry context.

4Why is timeout not proof that a tool action failed?

The remote action may have succeeded even though the response was lost, so the outcome can be uncertain.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.