Gnu Group Learning AcademyGnu Group ↗
SRE / Reliability Engineering Fundamentals · Module 04 of 12

Monitoring signals and observability

Use black-box and white-box evidence, the four golden signals and symptom-versus-cause reasoning to build monitoring that supports reliability decisions instead of accumulating dashboards.

Running case · checkout service

Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.

Learning objectives

By the end of this module, you should be able to:

  • Differentiate black-box and white-box monitoring and use them together.
  • Apply latency, traffic, errors and saturation to a user-facing service.
  • Separate symptoms that justify action from diagnostic causes that help explain them.

Monitor symptoms and causes differently

A page should normally tell the on-call engineer that a user-relevant symptom requires action. CPU, queue depth or a failed dependency may explain the symptom, but those causes should not automatically page humans when users remain unaffected.

This separation reduces noise while preserving diagnostic depth.

The four golden signals are a useful starting frame

Latency, traffic, errors and saturation provide broad coverage for request-serving systems. They are not a mandatory dashboard template; adapt them to the service and user journey.

Distinguish successful-request latency from fast failures, and treat saturation as capacity pressure rather than simply 100% resource utilization.

Observability supports questions you did not predefine

Metrics, logs and traces can be correlated to investigate unexpected behaviour. SRE uses these signals in service of reliability outcomes: detection, triage, capacity planning, change validation and post-incident learning.

  • black-box probes
  • service-level metrics
  • structured logs
  • distributed traces
  • change/deployment markers

Tail latency and cardinality can distort monitoring

Averages can look healthy while the p95 or p99 user experience is poor. Use distributions or threshold-based latency SLIs for user-facing decisions, then inspect percentiles as diagnostic context.

High-cardinality labels such as request IDs or unbounded user identifiers can make metrics expensive and unstable. Keep high-cardinality detail in logs/traces and use bounded dimensions in metrics.

RED and USE are complementary diagnostic frames

For request-serving services, RED asks about rate, errors and duration. For infrastructure resources, USE asks about utilization, saturation and errors.

Neither framework replaces a user SLI; they organize diagnostic evidence around the service-level symptom.

Worked example

Symptom versus cause

SYMPTOM: checkout good-event ratio below SLO
CAUSE CANDIDATES: DB connection errors · queue saturation · bad release
  • Page on the user-impacting symptom when urgent.
  • Use internal telemetry and change history to narrow the cause.
Core lab · account-free

Lab 04 — Classify a reliability evidence bundle

Goal: Classify synthetic signals into latency, traffic, errors and saturation, then identify symptom and likely causes.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
  • Running case study: use the same checkout journey throughout the course unless the lab explicitly says otherwise.
STEP 1

Create evidence

cat > /tmp/sre-signals.json <<'EOF'
{"request_rate":420,"p95_ms":880,"error_ratio":0.07,"cpu":0.52,"queue_depth":950,"queue_capacity":1000,"deploy":"2026-09-30T10:00:00Z"}
EOF
STEP 2

Calculate saturation

python3 - <<'PY2'
import json
x=json.load(open('/tmp/sre-signals.json'))
print('traffic_rps',x['request_rate'])
print('latency_p95_ms',x['p95_ms'])
print('errors_pct',x['error_ratio']*100)
print('queue_saturation_pct',x['queue_depth']*100/x['queue_capacity'])
PY2
STEP 3

Write the triage statement

State the user-visible symptom first, then list at least two cause hypotheses including the deployment marker.

Verify

  • The queue is identified as 95% saturated even though CPU is only 52%.
  • The triage statement does not mistake a suspected cause for the user-impacting symptom.

Expected outcome

  • A compact golden-signals view and a symptom-first diagnosis statement.

Diagnose / troubleshoot

  • If you page on every high resource metric, ask whether it predicts or represents user impact.
  • If a dashboard has no traffic context, you may misread latency/error changes at very low or very high load.
Real-world connection

Reliability monitoring should tell responders what is broken for users and provide enough context to investigate why.

Real-tool practicum · optional but recommended

Optional practicum — observe the local demo service

Generate success, failure and slow traffic, then inspect Prometheus query results.

Prerequisites

  • Docker/Compose available.
  • SRE Lab Pack or release source tree.

Resources created

  • Local demo service metrics including request counters and latency histogram.

Cost / risk

  • Local CPU/RAM only.
TOOL 1
./ops/setup-sre-local-lab.sh
TOOL 2
curl -s -o /dev/null http://127.0.0.1:8000/ok
TOOL 3
curl -s -o /dev/null http://127.0.0.1:8000/slow
TOOL 4
curl -s -o /dev/null http://127.0.0.1:8000/fail || true
TOOL 5
curl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=sum by (outcome) (increase(sre_demo_requests_total[5m]))'

Verify

  • Prometheus separates request outcomes and the demo exposes latency/error behaviour.

Cleanup

  • ./ops/cleanup-sre-local-lab.sh

Warnings

  • Keep labels bounded; do not add request IDs or user IDs to Prometheus metrics.
  • Use service-level symptoms for paging and infrastructure signals for diagnosis.
Avoid these traps

Common mistakes

  • Paging directly on every infrastructure threshold.
  • Using averages that hide tail latency.
  • Building dashboards with no explicit decision or troubleshooting purpose.
Knowledge check

Can you connect the reliability objective to an engineering decision?

Three concept checks plus one scenario check.

1What are the four golden signals?

Latency, traffic, errors and saturation.

2What is the main distinction between black-box and white-box monitoring?

Black-box monitoring observes externally visible behaviour; white-box monitoring uses internal system telemetry.

3Why separate symptoms from causes?

Symptoms express user impact and urgency, while causes are diagnostic hypotheses that can change without requiring a human page.

4Scenario: CPU is normal but a request queue is 98% full and p95 latency is rising. Which golden signal highlights the capacity risk?

Saturation; the queue is approaching its limit even though CPU is not.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.