LLMOps & AI Platform Engineering · Module 10 of 16
Observing AI Systems
Observability combines logs, metrics and traces so operators can ask both 'is the system healthy overall?' and 'what happened to this request?'. AI systems add useful context such as model/backend identity, token usage, queue/TTFT and evaluation signals, but telemetry must avoid becoming an uncontrolled copy of sensitive prompts and responses.
Engineering focus: Logs, metrics, traces, request correlation and AI-specific telemetry
In plain English
What problem are we solving?
Observability combines logs, metrics and traces so operators can ask both 'is the system healthy overall?' and 'what happened to this request?'. AI systems add useful context such as model/backend identity, token usage, queue/TTFT and evaluation signals, but telemetry must avoid becoming an uncontrolled copy of sensitive prompts and responses.
Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.
Animated mental model
Follow one trace from client through gateway and inference
Metrics tell you the population is slow; a trace shows where this request spent its time. Logs add detailed events when you need them.
● Loading interaction engine…
● PAUSEDTrace begins
01Client spanuser requestNOW
02Gateway spanpolicy + routeNOW
03Queue spanwaitingNOW
04Inference spanmodel workNOW
05Stream spanresponseNOW
06Metrics + logspopulation + eventsNOW
STEP 1/6Trace begins
The client creates or propagates trace context so downstream work can be connected to one request.
trace=7f2a…Learning objectives
By the end of this module, you should be able to:
- Explain the complementary roles of logs, metrics and traces.
- Propagate a request/trace identity across gateway and inference boundaries.
- Choose AI-serving telemetry such as TTFT, queue, token usage, backend/model and errors without indiscriminate content logging.
- Use a trace plus aggregate metrics to localize a production symptom.
🎓 Learning Coach follows this block when it is centered in your browser
Metrics, traces and logs answer different questions
Metrics summarize behavior over time: request rate, error rate, queue depth, latency percentiles, token usage or GPU/cache pressure. Traces follow individual requests across components. Logs record discrete diagnostic events and details.
A strong investigation often starts with metrics to find the time window, uses a trace to localize a slow request, then inspects structured logs/events for detailed failure information.
- Metrics: trends, rates, distributions and alerting.
- Traces: causal path/timing for individual requests.
- Logs/events: detailed discrete occurrences and diagnostics.
- Correlation IDs connect signals across components.
Common beginner misconception
Centralized logs alone are not full observability if you cannot quantify service health or follow request causality.
🎓 Learning Coach follows this block when it is centered in your browser
AI serving adds model, token and phase context to normal service telemetry
Traditional HTTP metrics still matter: rate, errors and duration. AI systems add workload dimensions such as logical model/backend, input/output token counts, TTFT, queue/prefill/decode timing and sometimes cache utilization.
Current vLLM serving exposes a Prometheus-compatible /metrics endpoint and supports OpenTelemetry trace export. Its observability configuration includes options for detailed traces and KV-cache-related metrics, with warnings that detailed timing can have performance cost.
- Request rate/error/status remain foundational.
- Queue depth and TTFT help identify pre-generation saturation.
- Input/output token counts describe work shape and cost.
- Model/backend/version attributes connect incidents to releases/routes.
Common beginner misconception
Token count is useful context, but it does not prove answer quality. Observability and evaluation remain separate evidence layers.
🎓 Learning Coach follows this block when it is centered in your browser
OpenTelemetry provides a vendor-neutral trace/metric/log model, but conventions evolve
OpenTelemetry defines common telemetry data structures and semantic conventions so instrumentation can be exported to different backends. HTTP semantic conventions are stable in current OpenTelemetry documentation, while GenAI conventions are evolving separately and should be version-aware.
Trace context must propagate through the gateway to the inference server for one end-to-end trace. vLLM currently documents OTLP trace export and an OpenTelemetry proof-of-concept path.
- Trace = end-to-end request tree; span = timed operation within it.
- Context propagation keeps downstream spans attached to the same trace.
- Resource/service attributes identify which component/version emitted telemetry.
- Semantic convention versions should be treated as an integration contract.
Common beginner misconception
Adding OpenTelemetry libraries does not automatically give a useful trace; boundaries and context propagation still need correct instrumentation.
🎓 Learning Coach follows this block when it is centered in your browser
Telemetry must be useful without becoming a data leak or cost explosion
Prompts, retrieved documents and model responses may contain personal, confidential or regulated data. Recording them by default in traces/logs can create a second uncontrolled data store. OpenTelemetry GenAI guidance warns that several content attributes may contain sensitive information.
Metrics also need cardinality discipline. Labels such as raw user ID, prompt text or request ID can create huge time-series counts. Put per-request identity in traces/logs; keep metric dimensions bounded.
- Default to metadata over raw content where possible.
- Redact/hash identifiers only when the resulting value still has a justified operational purpose.
- Keep metrics dimensions bounded: model, route, status, tenant tier—not raw request IDs.
- Define retention and access controls for telemetry itself.
Common beginner misconception
Hashing sensitive data does not automatically make it safe or low-cardinality.
Interactive simulator
Trace viewer: diagnose normal, queue-spike and provider-timeout requests
Switch scenarios and inspect span duration, status, model/backend attributes and the correlated metrics panel.
● Loading simulator engine…
This simulator teaches relationships and diagnostic reasoning. Its synthetic values are not vendor benchmarks, hardware-sizing evidence or production SLO measurements.
Worked examples
One request trace
trace REQ-18427
client 0.00 ─ 4.60s
gateway 0.02 ─ 4.55s
queue 0.05 ─ 3.95s ← 3.90s
inference 3.95 ─ 4.50s ← 0.55s
stream 4.50 ─ 4.60s
metric: p95_ttft=4.1s, queue_depth=72- The trace localizes this request's delay to queueing.
- Aggregate metrics show the problem is not isolated to one request.
Bounded metric labels
GOOD metric labels
model=small-chat
backend=vllm-a
status=ok
tenant_tier=standard
BAD metric labels
request_id=REQ-18427
prompt="full customer message..."- Bounded labels support aggregation.
- Unique/sensitive values belong in controlled trace/log context when justified, not metric dimensions.
Try it · core lab · account-free
Correlate synthetic metrics, traces and structured events
Goal: Diagnose three AI-serving incidents by combining aggregate and request-level evidence.
Before you start
- Python 3 only. The lab emits JSON lines and a text trace; no telemetry backend is required.
Inspect a healthy request
python3 public/labs/llmops/10_observability_trace_lab.py healthyInspect queue saturation
python3 public/labs/llmops/10_observability_trace_lab.py queue-spikeInspect provider timeout
python3 public/labs/llmops/10_observability_trace_lab.py provider-timeoutCompare signals
For each scenario, identify what the metric says globally, what the trace localizes and what the event/log explains.
Verify
- Healthy scenario reports diagnosis=healthy.
- Queue scenario reports diagnosis=queue_pressure with a dominant queue span.
- Timeout scenario reports diagnosis=provider_timeout and an error event correlated to the same trace.
Expected outcome
- You can explain why no single telemetry signal is sufficient for every investigation.
- You can distinguish safe bounded metric labels from high-cardinality/sensitive content.
Break / fix · diagnose
- If you diagnose only from the log message, compare whether metrics show one request or a population-wide issue.
- If request_id appears as a metric dimension in your modification, move it to trace/log context.
Production view
Production telemetry should correlate gateway and inference spans, expose service/AI workload metrics, and deliberately control sensitive content and metric cardinality.
Production Engineer checklist
Before you call this concept production-ready:
This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.
Avoid these traps
Common mistakes
- Treating logs, metrics and traces as interchangeable.
- Logging complete prompts/responses by default for every request.
- Using request IDs or raw user IDs as metric labels.
- Collecting detailed tracing everywhere without measuring telemetry overhead or setting sampling/retention policy.
Knowledge check
Reason about the system, not just the vocabulary.
Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.
1Metrics show latency regression. Which signal best shows where one affected request spent its time?
A distributed trace with spans across gateway, queue and inference.
2Why should raw prompt text normally not be a metric label?
It is sensitive and effectively unbounded/high-cardinality, creating privacy and metric-system cost problems.
3What does trace-context propagation accomplish?
It keeps downstream gateway/inference spans attached to the same end-to-end request trace.
4Why are evaluation metrics still needed if infrastructure observability is excellent?
Operational telemetry can show health/performance but does not prove the semantic quality or task correctness of model answers.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.