Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 06 of 12

Logs and structured events

Turn application events into searchable, correlatable evidence using levels, timestamps, stable fields and safe data-handling practices.

Learning objectives

By the end of this module, you should be able to:

  • Prefer structured fields over unparseable prose for machine investigation.
  • Use severity levels consistently.
  • Avoid secrets, tokens and unnecessary personal data in logs.

Structure enables reliable filtering

Structured logs encode fields such as timestamp, level, service, event, route and trace_id. Free text remains useful as a message, but key dimensions should not require brittle parsing.

Severity should mean something

INFO describes normal significant events, WARN signals abnormal but handled conditions, and ERROR signals failed operations requiring attention. Teams should define their own conventions and avoid turning every event into an error.

Logs are a data-governance surface

Credentials, authorization headers, session tokens and unnecessary personal data should not be logged. Retention and access should match risk and operational need.

Worked example

See the investigation idea in practice.

Structured event

{"ts":"2026-09-29T09:15:00Z","level":"error","service":"checkout","event":"db_pool_timeout","trace_id":"bbb","route":"/checkout"}
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 06 — Parse and correlate structured logs

Goal: Filter error events and extract correlation identifiers.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create JSONL events

cat > /tmp/app.jsonl <<'EOF'
{"level":"info","event":"request_ok","trace_id":"aaa"}
{"level":"error","event":"db_pool_timeout","trace_id":"bbb"}
{"level":"info","event":"request_ok","trace_id":"ccc"}
EOF
STEP 2

Filter errors

python3 - <<'PY'
import json
for line in open('/tmp/app.jsonl'):
 e=json.loads(line)
 if e['level']=='error': print(e['event'], e['trace_id'])
PY
STEP 3

Review safety

Confirm no credentials, access tokens or unnecessary personal data are present.

Verify

  • Only `db_pool_timeout bbb` is selected as an error event.
  • The trace ID is available for correlation.

Expected outcome

  • A machine-searchable event model with useful context and no sensitive secret material.

Diagnose / troubleshoot

  • If JSON parsing fails, validate one event per line and correct quoting.
  • If correlation is impossible, add a stable trace/request identifier at instrumentation time.
Real-world connection

Structured events reduce time spent reverse-engineering ad-hoc log formats during incidents.

Optional real-tool lab

Optional tool lab — emit structured Collector logs

Send an OTLP log record to the local Collector and inspect its debug output.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose up -d
TOOL 3
./send-otlp-log.sh
TOOL 4
docker compose logs otel-collector --tail=80

Verify

  • Collector logs show the sample log record or its attributes after OTLP ingestion.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Logging secrets or authorization headers.
  • Using ERROR for routine expected conditions.
  • Embedding all useful fields inside one free-text message.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1Why prefer structured logs?

They make important fields consistently filterable and correlatable.

2What is a correlation ID for?

Connecting related evidence across events and often across telemetry signals.

3Why avoid secrets in logs?

Logs are widely copied/retained and can expose credentials or sensitive information.

4Scenario: responders can see an exception message but cannot join it to a trace. What instrumentation field is most helpful?

A stable trace ID or request correlation identifier emitted in both signals.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.