AI Engineering Fundamentals · Module 10 of 12

See what the AI system is doing

Trace retrieval, model and tool steps; measure latency, errors, fallbacks, usage and quality signals; and collect enough telemetry to diagnose failures without unnecessarily logging sensitive content.

Engineering focus: AI observability and operations

In plain English

What problem are we solving?

The rest of this module introduces the formal terminology and production controls behind that idea. You are not expected to memorize the vocabulary before using it.

Running case · Atlas Assist

Carry the same support assistant forward: approved KB, source citations, explicit insufficient-evidence fallback and one bounded read-only asset lookup. Reuse prior artifacts so each module adds evidence to one production AI system.

Learning objectives

By the end of this module, you should be able to:

Define telemetry across retrieval, model and tool stages.
Separate operational health metrics from AI quality metrics.
Instrument without defaulting to full prompt/response logging.

Trace the application, not only the model call

A useful trace connects request intake, retrieval, context assembly, model inference, tool calls, validation and final response. Otherwise latency or quality failures cannot be localized.

Operational and quality signals are different

Availability, p95 latency, token throughput and tool error rate show whether the service is functioning. Groundedness, task success, abstention correctness and user feedback show whether it is useful.

Cost and capacity are first-class telemetry

Track input/output tokens or equivalent workload measures, cache hit rates, model routing, concurrency and per-feature usage. A quality improvement that multiplies cost may still violate the product contract.

Semantic conventions reduce telemetry fragmentation

OpenTelemetry semantic conventions provide common naming patterns, including evolving conventions for GenAI systems. Adopt stable/approved attributes deliberately and pin semantic-convention versions where compatibility matters.

Content telemetry creates privacy risk

Do not assume traces should capture raw prompts, retrieved chunks or responses. Prefer IDs, hashes, lengths, classifications and sampled/redacted content when needed for debugging under an explicit policy.

Alert on actionable service outcomes

Page on symptoms that require timely human action—sustained failures, tool outage, latency SLO burn or policy-control failure—not on every unusual model response.

Worked examples

AI trace

request
 ├─ retrieval(query, top_k, source_ids, ms)
 ├─ model(model_id, input_units, output_units, ms)
 ├─ tool(name, authorized, result, ms)
 └─ validator(status, citations, policy)
→ outcome(task_success/fallback)

One trace explains where time and decisions occurred.

Two dashboards

OPERATIONS: rate · errors · latency · saturation · cost
QUALITY: task success · groundedness · retrieval recall · fallback correctness · safety slices

Do not let an uptime dashboard masquerade as AI quality.

Try it · core lab · account-free

Lab 10 — Design an AI telemetry event

Goal: Create privacy-minimized trace-shaped records and derive operational metrics.

Before you start

No paid API or cloud account is required for the core lab.
Core labs use Python 3 and local files only.
Running case: you are engineering 'Atlas Assist', an internal support assistant over a small approved knowledge base with one read-only inventory tool.
Carry requirements, eval cases, threat assumptions and operational evidence forward across modules instead of starting over.

STEP 1

Create sample telemetry

cat > /tmp/ai-telemetry.jsonl <<'EOF'
{"trace":"t1","model":"local-small","latency_ms":820,"input_units":900,"output_units":120,"retrieved":3,"status":"ANSWERED","tool_calls":0}
{"trace":"t2","model":"local-small","latency_ms":1450,"input_units":1600,"output_units":40,"retrieved":0,"status":"INSUFFICIENT_EVIDENCE","tool_calls":0}
EOF

STEP 2

Aggregate without prompt content

python3 - <<'PY2'
import json
r=[json.loads(x) for x in open('/tmp/ai-telemetry.jsonl')]
print('requests',len(r),'avg_ms',sum(x['latency_ms'] for x in r)/len(r),'fallbacks',sum(x['status']!='ANSWERED' for x in r))
assert all('prompt' not in x and 'response' not in x for x in r)
print('PASS: privacy-minimized operational telemetry')
PY2

STEP 3

Add quality linkage

Add eval_dataset_version and application_version IDs so production samples can be compared with the correct offline baseline without storing raw sensitive text by default.

Verify

The telemetry localizes model/retrieval/tool stages without requiring raw prompt content.
Operational and quality metrics are represented separately.

Expected outcome

A privacy-conscious telemetry schema and basic operational aggregation.

Break / fix · diagnose

If traces contain secrets or full documents, add minimization/redaction before increasing retention.
If a model error alert has no required human action, route it to a dashboard or ticket instead of paging.

Production view

AI production incidents become diagnosable when model, retrieval, tools and application policy share one trace context.

Local real-tool practicum · optional but recommended

Optional practicum — inspect Ollama timing metadata

Capture request duration/token counters and turn them into a minimal local telemetry record.

Prerequisites

Ollama installed and reachable on http://127.0.0.1:11434 for the real-tool path.

Resources created

Only local disposable files/processes used by the practicum.

Cost / risk

No cloud account required; model downloads consume disk and network bandwidth.

TOOL 1

MODEL=${MODEL:-qwen3:8b}; curl -s http://127.0.0.1:11434/api/chat -H 'Content-Type: application/json' -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Say observability in five words.\"}],\"stream\":false}" > /tmp/ollama-telemetry.json

TOOL 2

python3 - <<'PY2'
import json
x=json.load(open('/tmp/ollama-telemetry.json'))
for k in ['model','total_duration','prompt_eval_count','eval_count']: print(k,x.get(k))
PY2

Verify

The record exposes model identity plus duration and token/evaluation counters without printing the response content.

Cleanup

Remove only the disposable files created for the lab; keep model caches only if you intend to reuse them.

Warnings

Telemetry field names vary by provider/runtime. Normalize into an application schema and document units.

Avoid these traps

Common mistakes

Monitoring only model uptime.
Mixing quality and availability into one ambiguous KPI.
Logging full prompts by default.
Ignoring model routing/token usage when investigating cost regressions.

Knowledge check

Can you connect the AI behaviour to a system control?

Concept and scenario checks. Reveal the answer only after you commit to a decision.

1Why trace retrieval and tools as well as inference?

Because latency, correctness and policy failures can occur in any stage and need attribution.

2What is the difference between service health and AI quality?

Health measures operational functioning; quality measures whether outputs satisfy the user and evidence contract.

3Why minimize prompt content in telemetry?

Prompts/retrieved text can contain secrets, personal data or confidential information and create unnecessary retention risk.

4Scenario: latency doubles after a model change while answer quality holds. What evidence should be compared?

Model identity, prompt/context units, inference timing, routing/cache and concurrency telemetry for baseline versus candidate.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.