Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 09 of 12

Correlating metrics, logs and traces

Use shared dimensions, timestamps and trace identifiers to move from service-wide symptoms to one concrete failing request and back again.

Learning objectives

By the end of this module, you should be able to:

  • Pivot from a metric symptom to logs and traces.
  • Use time, service and trace identifiers to maintain investigation scope.
  • Avoid correlation workflows that depend on copying unrelated search terms manually.

Start broad, then narrow

Metrics can identify when and where a population changed. Logs add event context. Traces show one request path. Correlation is the deliberate pivot between these views while preserving time/service/request context.

Exemplars and trace IDs reduce guesswork

A metric exemplar can link an observation to a trace. A trace ID in structured logs links request-level events to the same execution path.

Correlation should challenge hypotheses

Use another signal to falsify the leading hypothesis, not merely to collect confirming screenshots.

Worked example

See the investigation idea in practice.

Cross-signal pivot

metric: checkout error ratio 7% @ 09:15
log: db_pool_timeout trace_id=bbb @ 09:15
trace bbb: db.pool.wait=1700ms
resource: app CPU=42%, db_pool=98%
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 09 — Correlate one incident

Goal: Join metric, log and trace evidence into a single timeline and reject one competing hypothesis.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create evidence files

printf '%s\n' '09:15,error_ratio,0.07' '09:15,db_pool_pct,98' '09:15,app_cpu_pct,42' > /tmp/m.csv
printf '%s\n' '09:15 error db_pool_timeout trace_id=bbb' > /tmp/l.log
printf '%s\n' 'bbb checkout 2200' 'bbb db.pool.wait 1700' > /tmp/t.txt
STEP 2

Build a timeline

paste -d' | ' /tmp/m.csv /tmp/l.log /tmp/t.txt 2>/dev/null || true
STEP 3

Reject one hypothesis

State why CPU saturation is weakly supported while DB-pool saturation is strongly supported.

Verify

  • Evidence cites metric, log and trace data.
  • At least one competing hypothesis is explicitly rejected with counter-evidence.

Expected outcome

  • A defensible diagnosis that explains multiple signals and contradictory evidence.

Diagnose / troubleshoot

  • If timestamps use different zones, normalize them before correlating.
  • If IDs do not propagate across services, fix instrumentation rather than relying on fuzzy text matching.
Real-world connection

Correlation reduces mean time to understand by turning separate telemetry stores into one investigation workflow.

Optional real-tool lab

Optional tool lab — correlate Collector output

Use the sample OTLP trace and log scripts with the same trace identifier.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose up -d
TOOL 3
./send-otlp-trace.sh
TOOL 4
./send-otlp-log.sh
TOOL 5
docker compose logs otel-collector --tail=180 | grep -E "bbb|db_pool_timeout|db.pool.wait"

Verify

  • At least two telemetry records share the intended correlation context.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Collecting only confirmatory evidence.
  • Ignoring time-zone/skew differences.
  • Assuming same timestamp means causal relationship.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1What is a useful correlation pivot?

Move from a metric symptom to a trace/log using shared time, service, route or trace ID context.

2Why preserve trace IDs in structured logs?

They connect event context to the distributed request path.

3Why seek counter-evidence?

To reduce confirmation bias and test whether the leading hypothesis really explains the incident.

4Scenario: errors rise, a timeout log names trace `bbb`, and that trace spends 1.7s waiting for DB pool. What is the strongest next hypothesis?

Database connection-pool saturation or exhaustion.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.