Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 10 of 12

Alerting and incident diagnosis

Design alerts that are actionable, symptom-oriented and routed with enough context to begin diagnosis rather than merely announce that a threshold changed.

Learning objectives

By the end of this module, you should be able to:

  • Distinguish page-worthy symptoms from diagnostic signals.
  • Use duration/for semantics to reduce transient noise.
  • Attach ownership, severity and runbook context to alerts.

Alert on impact, diagnose with causes

A page should normally represent user/system impact that needs timely human action. CPU, memory or queue depth can be useful diagnostic signals without always being page-worthy by themselves.

Duration reduces flapping

A condition that must remain true for a duration can avoid alerting on brief noise. The duration must still respect how quickly the failure can harm users.

An alert is an operational contract

Useful alerts identify service, environment, severity, symptom, current value/threshold, ownership and a next diagnostic action or runbook.

Worked example

See the investigation idea in practice.

Actionable alert

CheckoutHighErrorRatio
condition: error_ratio > 0.02 for 10m
labels: severity=page team=commerce
annotations: impact, dashboard, runbook, current value
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 10 — Classify alerts

Goal: Separate page, ticket and dashboard-only signals for a small service.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Create candidates

cat > /tmp/alerts.csv <<'EOF'
signal,value
checkout_error_ratio,0.07
app_cpu_pct,42
db_pool_pct,98
EOF
STEP 2

Assign response

For each signal choose page, ticket, or diagnostic-only and justify using user impact and urgency.

STEP 3

Add alert context

Write owner, severity, dashboard and runbook for the page-worthy condition.

Verify

  • The user-impact symptom drives paging.
  • Resource saturation is retained as diagnostic evidence even if it is not the primary page.

Expected outcome

  • An alert set that minimizes noise while giving responders immediate scope and ownership.

Diagnose / troubleshoot

  • If an alert fires repeatedly without action, decide whether threshold, duration, ownership or alert purpose is wrong.
  • If responders need another dashboard just to learn which service is affected, add that context to the alert.
Real-world connection

Alert fatigue is often a design problem, not a discipline problem.

Optional real-tool lab

Optional tool lab — validate rules with promtool

Run Prometheus rule/config validation inside the pinned local image.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
cd labs/observability-stack
TOOL 2
docker compose run --rm prometheus promtool check config /etc/prometheus/prometheus.yml
TOOL 3
docker compose run --rm prometheus promtool check rules /etc/prometheus/rules.yml

Verify

  • Prometheus configuration and alert rule files pass `promtool` validation.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Avoid these traps

Common mistakes

  • Paging on every high CPU sample.
  • No owner/runbook on urgent alerts.
  • Using very long `for` durations that hide fast user impact.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1What usually makes a good paging alert?

A sustained user/system-impact symptom that requires timely human action.

2Why use a `for` duration?

To require a condition to persist and reduce transient/flapping alerts.

3What context should an alert include?

Service/scope, severity, ownership, symptom/value and a diagnostic/runbook path.

4Scenario: checkout errors are 7%, app CPU 42%, DB pool 98%. What should page?

The sustained checkout error symptom; DB-pool saturation is strong diagnostic evidence for the incident.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.