Gnu Group Learning AcademyGnu Group ↗
Observability Fundamentals · Module 12 of 12

Capstone — production incident diagnosis

Investigate a production-shaped checkout incident from synthetic metrics, logs, traces and an alert; reject a plausible wrong hypothesis, propose mitigation and prove recovery with numeric evidence.

Learning objectives

By the end of this module, you should be able to:

  • Correlate multiple telemetry signals into a defensible diagnosis.
  • Reject at least one competing hypothesis with counter-evidence.
  • Define safe mitigation, rollback and measurable recovery criteria.

The capstone is an evidence argument

The goal is not to guess the hidden answer. Build a chain from symptom → hypotheses → evidence → diagnosis → mitigation → verification.

Relationships are validated

The manifest requires an `evidence_ids` array from multiple signal types, a rejected hypothesis, control mapping, numeric recovery thresholds and follow-up improvements. Boolean self-attestation is not sufficient.

Recovery must be observable

A mitigation is incomplete until responders define what healthy means after the change: error ratio, p95 latency, pool utilization and alert state in this scenario.

Worked example

See the investigation idea in practice.

Incident evidence

M1 error_ratio=7%
M2 p95=2200ms
M3 app_cpu=42%
M4 db_pool=98%
L1 db_pool_timeout trace_id=bbb
T1 checkout→db.pool.wait=1700ms
A1 high_error_ratio FIRING
  • Read the evidence before deciding what it means.
  • Prefer a hypothesis that explains multiple independent signals.
Core lab · account-free

Lab 12 — Diagnose, mitigate and prove recovery

Goal: Create a relationship-aware incident report that the validator checks against the evidence pack.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
STEP 1

Inspect evidence

cat labs/observability-capstone/evidence.json
STEP 2

Create your report

cp labs/observability-capstone/report.example.json /tmp/observability-report.json
$EDITOR /tmp/observability-report.json
STEP 3

Run relationship validator

python3 labs/observability-capstone/validate_report.py /tmp/observability-report.json
STEP 4

Test a bad hypothesis

Change `primary_diagnosis` to `application_cpu_saturation`; the validator must reject it.

STEP 5

Restore and prove recovery

Restore the evidence-supported diagnosis and verify all numeric recovery thresholds and follow-up fields pass.

Verify

  • Validator requires metric + log + trace/alert evidence.
  • `application_cpu_saturation` is rejected because CPU is only 42%.
  • RTO/recovery-style verification thresholds are numeric and internally consistent.

Expected outcome

  • Diagnosis: database connection-pool saturation.
  • Mitigation is rollback-ready.
  • Recovery requires error ratio <=2%, p95 <=750ms, DB pool <=85% and alert clear.

Diagnose / troubleshoot

  • If validator says evidence class missing, add a cited metric/log/trace/alert ID rather than another prose claim.
  • If a rejected hypothesis has no counter-evidence, cite the evidence that makes it weaker.
  • If verification is vague (“looks good”), replace it with numeric thresholds.
Real-world connection

Production incident reviews are stronger when the diagnosis can be replayed from durable evidence and explicit reasoning.

Optional real-tool lab

Optional tool lab — run the full local stack validator

Exercise pinned Prometheus, Grafana and OpenTelemetry Collector health plus sample OTLP ingestion.

Resources created

  • Local containers/files only; no cloud resources.

Cost / risk

  • Expected cloud cost: none. Local CPU, memory and disk usage only.

Prerequisites

  • Docker Engine with Compose plugin for the optional shared stack.
  • Core course completion does not require Docker or external credentials.
TOOL 1
npm run labs:observability:stack

Verify

  • Prometheus and Grafana health checks pass.
  • Prometheus self target is UP.
  • Collector accepts sample OTLP telemetry and emits debug evidence.
  • Stack is cleaned up after validation.

Cleanup

  • Run `docker compose down -v` from `labs/observability-stack`.

Warnings

  • Optional only: not required by the account-free validator.
  • The bundled local stack is for training, not a production observability architecture.
Capstone rubric

Score evidence, not confidence.

Symptom and scope · 15%

States the user-visible symptom, affected service and time window without prematurely asserting a cause.

Multi-signal evidence · 25%

Cites metric, log and trace/alert evidence with identifiers.

Diagnosis and counter-evidence · 20%

Chooses a diagnosis that explains evidence and explicitly rejects at least one competing hypothesis.

Mitigation and rollback · 15%

Provides a safe mitigation with rollback readiness and ownership.

Verification · 15%

Defines numeric post-change thresholds and alert-clear criteria.

Follow-up · 10%

Adds at least two durable instrumentation/reliability improvements.

Avoid these traps

Common mistakes

  • Choosing root cause from one chart.
  • Writing “fixed” without numeric verification.
  • No rollback plan for mitigation.
  • Treating CPU correlation as causation despite counter-evidence.
Knowledge check

Can you reason from the signal?

Three concept checks plus one scenario check. Open each answer only after deciding first.

1What makes a diagnosis defensible?

It explains multiple independent signals and survives competing-hypothesis testing.

2Why require a rejected hypothesis?

To demonstrate counter-evidence and reduce confirmation bias.

3What should mitigation include?

A safe action, owner/context and rollback readiness.

4Scenario: errors 7%, p95 2200ms, CPU 42%, DB pool 98%, timeout log and 1700ms pool-wait span. What diagnosis best fits?

Database connection-pool saturation.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.