Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 01 of 12

Observability mental model and signals

Build the mental model: observability uses telemetry and system context to answer questions about internal behavior, especially when the failure was not predicted in advance.

Learning objectives

By the end of this module, you should be able to:

  • Distinguish monitoring from observability without treating them as opposites.
  • Explain metrics, logs and traces as complementary signals.
  • Start an investigation from a user symptom and test hypotheses with evidence.

Monitoring asks known questions; observability supports investigation

Monitoring is the deliberate checking of known conditions such as availability, latency or saturation. Observability is the capability to infer internal system behavior from emitted evidence and context.

A dashboard is not observability by itself. Instrumentation, useful dimensions, correlation identifiers and an investigation workflow make telemetry actionable.

Three signals, one incident

Metrics summarize numeric behavior over time, logs record discrete events, and traces connect work across service boundaries.

No signal is universally best. Begin with the symptom, form a hypothesis, and use the cheapest useful signal to narrow the search.

  • Metrics: rate, latency, errors, saturation
  • Logs: structured events and exception context
  • Traces: request path, spans, parent/child timing

Evidence is not diagnosis

A CPU spike can coincide with a failure without causing it. Treat every signal as evidence whose meaning depends on timing, scope and system design.

Worked example

See the investigation idea in practice.

Symptom → hypothesis → evidence

User: checkout is slow
Hypothesis A: app CPU saturation
Hypothesis B: database pool saturation
Evidence: CPU 42%, DB pool 98%, p95 2.2s, db_pool_timeout logs
Decision: B explains more evidence
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 01 — Triage a synthetic incident

Goal: Use multiple signals to distinguish symptom, evidence and diagnosis.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create evidence

cat > /tmp/obs-incident.csv <<'EOF'
signal,value
error_ratio,0.07
p95_ms,2200
app_cpu_pct,42
db_pool_pct,98
EOF
cat /tmp/obs-incident.csv
STEP 2

Form two hypotheses

Write one application hypothesis and one dependency hypothesis before reading the next evidence.

STEP 3

Add event evidence

printf '%s\n' 'level=error event=db_pool_timeout trace_id=bbb' > /tmp/obs-event.log
cat /tmp/obs-event.log

Verify

  • You can state the user-visible symptom separately from the probable cause.
  • Your diagnosis cites at least two independent pieces of evidence.

Expected outcome

  • Database-pool saturation is better supported than application CPU saturation.

Diagnose / troubleshoot

  • If you jump directly from one metric to a root cause, write an alternative hypothesis first.
  • If evidence conflicts, keep the diagnosis provisional and collect another signal.
Real-world connection

Real incident response improves when teams separate observation, hypothesis, evidence and action.

Optional real-tool lab

Optional tool lab — open the local telemetry stack

Verify Prometheus and Grafana health before later modules.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose up -d
TOOL 3
curl -fsS http://127.0.0.1:9090/-/ready
TOOL 4
curl -fsS http://127.0.0.1:3001/api/health

Verify

  • Prometheus returns ready.
  • Grafana reports a healthy database.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Equating dashboards with observability.
  • Treating correlation as causation.
  • Collecting every possible signal without an investigation purpose.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1What is observability for?

Inferring internal system behavior from emitted evidence and context so engineers can investigate known and novel failures.

2How do metrics and logs differ?

Metrics summarize numeric behavior over time; logs preserve discrete event context.

3What does a trace add?

A trace connects spans across a request path and shows where time or errors accumulate.

4Scenario: p95 latency rises while CPU stays normal and DB-pool utilization reaches 98%. What should you investigate first?

Database connection-pool saturation, while keeping the diagnosis evidence-based rather than assuming CPU is causal.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.