Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 08 of 12

Distributed tracing

Follow one request across services using traces, spans, parent-child relationships, attributes and status so latency and errors can be localized.

Learning objectives

By the end of this module, you should be able to:

  • Explain trace and span relationships.
  • Interpret critical-path latency.
  • Recognize sampling trade-offs and avoid putting secrets in span attributes.

A trace is a request story

A trace groups spans that represent units of work. Parent-child relationships describe call structure, while timestamps and durations show where time was spent.

Critical path matters more than span count

The slowest request path is not necessarily the service with the most spans. Focus on spans that block user-visible completion and on dependency waits.

Sampling is an evidence trade-off

Head and tail sampling reduce telemetry volume but can discard useful incidents if configured poorly. Sampling policy should preserve enough evidence for reliability/security goals.

Worked example

See the investigation idea in practice.

Trace waterfall

checkout 2200ms
├─ auth 45ms
├─ inventory 90ms
└─ db.pool.wait 1700ms
   └─ db.query 240ms
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 08 — Find the critical path

Goal: Parse a synthetic trace and identify the dominant blocking span.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create span data

cat > /tmp/spans.csv <<'EOF'
span,parent,duration_ms
checkout,,2200
auth,checkout,45
inventory,checkout,90
db.pool.wait,checkout,1700
db.query,db.pool.wait,240
EOF
STEP 2

Find the longest child

python3 - <<'PY'
import csv
r=list(csv.DictReader(open('/tmp/spans.csv')))[1:]
print(max(r,key=lambda x:int(x['duration_ms'])))
PY
STEP 3

Connect to hypothesis

Explain why a long `db.pool.wait` span supports pool saturation more directly than an application CPU hypothesis.

Verify

  • `db.pool.wait` is identified as the dominant child span.
  • You distinguish wait time from actual database-query time.

Expected outcome

  • A trace-based localization of where the request is blocked.

Diagnose / troubleshoot

  • If child durations exceed parent wall-clock duration when summed, remember siblings can overlap.
  • If spans are missing, inspect context propagation and sampling before concluding the service was not called.
Real-world connection

Tracing is especially valuable when latency spans multiple services and queues.

Optional real-tool lab

Optional tool lab — ingest a trace

Send and inspect the bundled sample trace through the Collector.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose up -d
TOOL 3
./send-otlp-trace.sh
TOOL 4
docker compose logs otel-collector --tail=120 | grep -E "checkout|db.pool.wait|academy-checkout"

Verify

  • Collector output provides evidence for the sample trace/service.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Summing all child spans as if none overlap.
  • Adding high-cardinality/sensitive attributes without need.
  • Treating one sampled trace as a statistically complete latency view.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1What is a span?

A timed unit of work within a trace, with context and attributes.

2What does parent-child relationship show?

The causal/call structure connecting work within the request.

3What is a critical-path span?

Work whose latency directly blocks completion of the observed request path.

4Scenario: query spans are 200ms but `db.pool.wait` is 1700ms. Where is the dominant latency?

Waiting for a database connection, not executing the query itself.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.