SRE / Reliability Engineering Fundamentals · Module 08 of 12

Troubleshooting, runbooks and automation

Use hypothesis-driven troubleshooting, change correlation, layered evidence and executable runbooks to shorten incidents without turning procedural documents into brittle magic.

Running case · checkout service

Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.

Learning objectives

By the end of this module, you should be able to:

Apply symptom-first, hypothesis-driven troubleshooting to a distributed service.
Design runbooks around evidence, safe actions, verification and escalation.
Recognize when a repeated runbook sequence should become automated remediation.

Start with scope and recent change

Confirm the user-visible symptom, affected population, start time and whether a deployment, configuration or dependency change correlates with it. This prevents random command execution.

Build hypotheses that can be disproved by evidence instead of collecting every metric available.

Move through layers deliberately

A practical path is request/edge → service → dependency → runtime/resource → network/storage as evidence warrants. The order is not rigid; it exists to keep investigation connected to the symptom.

Correlate metrics, logs, traces and change events by time and request identity when available.

A runbook is a decision aid

Good runbooks describe when the procedure applies, prerequisites, safe commands, expected evidence, rollback and escalation. They should not hide dangerous assumptions.

When the same runbook executes predictably and safely, consider automating the decision or remediation with guardrails.

Runbooks need verification and rollback, not just commands

A production runbook should state scope, prerequisites, evidence to collect, safe action, expected result, rollback/abort condition, escalation and post-action verification.

Automation is appropriate only when the decision boundary can be expressed safely. Use dry-run modes, bounded blast radius, idempotent actions, rate limits and a human approval gate when uncertainty is high.

Worked example

Hypothesis loop

Symptom → hypothesis → smallest discriminating check → evidence → keep/reject → next hypothesis

Prefer a check that distinguishes between hypotheses.
Record what you learned so repeated incidents become faster to diagnose.

Runbook skeleton with guardrails

WHEN: checkout SLI below objective after release
CHECK: confirm release correlation + DB healthy
ACTION: rollback one release
VERIFY: checkout good-event ratio recovers for 10 min
ABORT: rollback worsens errors or integrity signal changes
ESCALATE: no recovery after rollback
AUTOMATION GUARD: only one rollback, signed artifact, recorded change ID

The runbook connects a user symptom to a bounded action and a measurable recovery check.
The automation guard prevents an endless remediation loop.

Core lab · account-free

Lab 08 — Diagnose a synthetic latency incident

Goal: Correlate a deployment marker, latency, error and queue data to rank hypotheses.

Before you start

No cloud account required.
Core lab uses bash/Python 3 and local files only.
Running case study: use the same checkout journey throughout the course unless the lab explicitly says otherwise.

STEP 1

Create timeline evidence

cat > /tmp/triage.csv <<'EOF'
time,p95_ms,error_pct,queue,deploy
09:55,180,0.2,80,no
10:00,190,0.3,85,yes
10:05,850,5.0,700,no
10:10,1200,9.0,940,no
EOF

STEP 2

Inspect the change and saturation trend

python3 - <<'PY2'
import csv
r=list(csv.DictReader(open('/tmp/triage.csv')))
print('deploy_at',[x['time'] for x in r if x['deploy']=='yes'])
print('max_queue',max(int(x['queue']) for x in r))
print('max_p95',max(int(x['p95_ms']) for x in r))
PY2

STEP 3

Write two competing hypotheses

For example: release increased work per request versus downstream slowdown. Name one evidence check that would discriminate between them.

Verify

The learner correlates, but does not automatically equate, the deployment with causation.
Each hypothesis has a specific evidence check.

Expected outcome

A symptom statement, ranked hypotheses and next discriminating checks.

Diagnose / troubleshoot

Correlation with a deployment is useful evidence but is not proof of cause.
Avoid changing multiple variables at once during diagnosis; it destroys evidence.

Real-world connection

Troubleshooting is fastest when each action is chosen to reduce uncertainty, not to demonstrate tool familiarity.

Real-tool practicum · optional but recommended

Optional practicum — use Linux evidence during a local incident

Inspect sockets/processes and query the demo service from the host.

Prerequisites

Docker/Compose plus standard Linux networking tools where available.

Resources created

Local demo service, logs and host-side socket/process evidence.

Cost / risk

Local CPU/RAM only.

TOOL 1

./ops/setup-sre-local-lab.sh

TOOL 2

curl -sv http://127.0.0.1:8000/slow -o /dev/null

TOOL 3

ss -ltn 2>/dev/null | grep -E ':8000|:9090' || true

TOOL 4

docker logs --tail 20 gnu-sre-demo

Verify

The responder can connect user-visible behaviour to a listening endpoint and service logs.

Cleanup

./ops/cleanup-sre-local-lab.sh

Warnings

Do not run copied diagnostic commands against production without understanding their scope.
Capture the symptom and hypothesis before changing multiple variables.

Avoid these traps

Common mistakes

Running a long checklist before defining the symptom.
Treating correlation with a recent change as proven causation.
Writing runbooks with commands but no verification or rollback.

Knowledge check

Can you connect the reliability objective to an engineering decision?

Three concept checks plus one scenario check.

1What is a strong first troubleshooting question?

What user-visible symptom exists, who is affected and when did it begin?

2What makes a hypothesis useful during an incident?

It predicts evidence that can distinguish it from competing explanations.

3When should a runbook step be considered for automation?

When it is repeated, predictable, safe to guard and provides durable value when automated.

4Scenario: latency rises five minutes after a deployment, but a dependency also reports saturation. What is the right conclusion?

Treat both as hypotheses and gather discriminating evidence; temporal correlation alone does not prove the deployment caused the incident.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.