Agentic AI Engineering · Module 12 of 16

Observability

When an agent fails, you need to know whether the model, a tool, policy, network, retrieval or approval caused the problem. Observability records safe events around the run so operators can reconstruct what happened.

Engineering focus: Tracing, metrics, audit events and cost/latency visibility

In plain English

What problem are we solving?

Framework names come later. First understand what the application is doing and where its control boundaries belong.

Running case · Operations Investigator

Across the course, imagine an assistant that investigates operational problems using narrow tools. The design progressively adds plans, specialist agents, approvals, reliability controls and MCP while keeping model output separate from execution authority.

Learning objectives

By the end of this module, you should be able to:

Design a trace that links model calls, tool calls, approvals and final outcome.
Choose useful metrics for success, latency, retries, tool failures and cost.
Log enough for diagnosis without storing secrets or hidden chain-of-thought.

Trace the system, not private reasoning

A useful trace records inputs at an appropriate redaction level, model/tool boundaries, decisions such as selected tool names, timings, result status, policy/approval outcomes and final task status. It does not require storing hidden chain-of-thought.

Correlation turns events into a run

Give each user task a run or correlation ID. Child spans/events for model calls, MCP/tool calls, approvals and retries inherit that identity so operators can follow one task across services.

Measure outcome and resource use

Useful measures include task success, tool success/error rate, p50/p95 latency, retry count, approval wait time, model tokens/cost, fallback usage and budget exhaustion. A fast agent that fails the task is not healthy.

Audit events are different from debug logs

For consequential actions, record who requested the task, which action was proposed, which policy decision was made, who approved, what was executed and the verification result. Audit events should be durable and tamper-resistant according to the organization’s needs.

Worked examples

One run, several spans

run_id=R42
  model.decide       180 ms
  tool.get_logs       35 ms
  model.decide       160 ms
  approval.wait      8.2 s
  tool.restart       420 ms
  tool.verify         40 ms
result=success

The trace shows where time was spent.
It also shows the approval boundary and verification step.

Safe structured event

{
  "run_id":"R42",
  "event":"tool_denied",
  "tool":"restart_service",
  "reason":"approval_required",
  "payload_logged":false
}

Structured fields are queryable.
Sensitive request bodies can remain omitted or redacted.

Try it · core lab · account-free

Build a safe agent run trace

Goal: Generate correlated events and summarize latency/failure information without logging sensitive payloads.

Before you start

Python 3 only.

STEP 1

Run the trace simulator

python3 public/labs/agentic-ai/12_observability.py

STEP 2

Follow one run ID

See model, tool, policy and verification events correlated to the same run.

STEP 3

Inspect redaction and metrics

Confirm the secret fixture never appears in event output and review the latency/error summary.

Download this Python lab Download all current labs

Verify

Every event includes the same run ID.
The secret string is absent from emitted telemetry.
The summary reports total latency, tool failures and retry count.

Expected outcome

You can localize a failure without reading hidden model reasoning.
You can explain the difference between operational telemetry and audit evidence.

Break / fix · diagnose

If events cannot be joined, propagate the run/correlation ID to every child operation.
If sensitive values appear, move to allowlisted structured fields and explicit redaction.

Production view

Production agents need distributed tracing plus outcome metrics and durable audit events. The exact telemetry schema can evolve, so pin the semantic conventions/version your platform adopts and test redaction.

Avoid these traps

Common mistakes

Logging every prompt and tool payload by default.
Measuring only model latency while ignoring tools and approval waits.
Treating a 200 HTTP response as task success.
Using logs without run/correlation identifiers.

Knowledge check

Explain the control decision, not just the vocabulary.

Open every answer only after committing to your own answer. Opening all four answers records knowledge-check activity in My Learning.

1Do you need hidden chain-of-thought to operate an agent safely?

No — trace observable system events, selected actions, tool/policy outcomes, timings and final task result instead.

2Why propagate a run ID across model and tool calls?

It lets operators reconstruct one end-to-end task across multiple services and retries.

3Which metric best complements latency for an agent?

Task success/outcome quality — low latency is not useful if the agent fails the job.

4What should an audit event record for an approved production change?

Requester/context, proposed action, policy result, approver/decision, execution result and verification evidence.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.