Gnu Group Learning AcademyGnu Group ↗
SRE / Reliability Engineering Fundamentals · Module 07 of 12

Incident response and severity

Coordinate incidents around impact, mitigation, roles and communication so responders reduce user harm quickly while preserving a clear timeline for later learning.

Running case · checkout service

Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.

Learning objectives

By the end of this module, you should be able to:

  • Classify incident severity using user/business impact rather than technical drama.
  • Separate incident command, operations, communications and subject-matter roles.
  • Prioritize mitigation and impact reduction before exhaustive root-cause analysis.

Severity is about impact

Severity should reflect user or business consequence, scope and urgency. A noisy technical failure with no user impact can be lower severity than a subtle data-integrity problem affecting a critical workflow.

Define severity criteria before an incident so teams do not negotiate them under stress.

Incident roles reduce coordination load

An incident commander coordinates the response, operations responders execute mitigation and diagnosis, and a communications role keeps stakeholders informed. Subject-matter experts can join without everybody trying to command the same incident.

Role separation is especially valuable as incidents cross team boundaries.

Mitigate first, diagnose deeply second

During acute user impact, a safe rollback, failover or feature disable can be preferable to proving the root cause before acting. Preserve evidence and hypotheses, but optimize first for reducing harm.

After stability returns, deeper causal analysis belongs in the post-incident process.

Use a published severity matrix and communication cadence

A severity scheme should define impact criteria, urgency, coordination expectations and communication cadence before an incident. The labels themselves are organization-specific.

Example: SEV-1 can mean widespread critical-journey unavailability, integrity/safety risk or severe business impact with immediate incident command; SEV-2 can represent significant partial degradation requiring urgent coordinated response; SEV-3 can cover localized impact handled by the owning team.

Worked example

Incident lifecycle

Detect → declare → assign roles → assess impact → mitigate → verify recovery → communicate → learn
  • Declaration creates a shared coordination state.
  • Recovery must be verified using user-facing signals, not assumed because a command succeeded.

Minimal stakeholder update

STATUS: Investigating / Mitigating / Monitoring / Resolved
IMPACT: who/what is affected
START: first known impact time
ACTION: current mitigation
NEXT UPDATE: explicit time or condition
  • State known impact and uncertainty without speculating about root cause.
  • Keep a predictable update cadence during major incidents so responders do not answer the same status question repeatedly.
Core lab · account-free

Lab 07 — Run a tabletop incident

Goal: Create an incident timeline, severity decision and mitigation plan from a synthetic production event.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
  • Running case study: use the same checkout journey throughout the course unless the lab explicitly says otherwise.
STEP 1

Review the event

Checkout error ratio rises to 35% immediately after a release. The rollback artifact is available and database health remains normal.

STEP 2

Classify and assign without a pre-filled answer

cat > /tmp/incident-response.json <<'EOF'
{"severity":"CHOOSE-SEV","commander":"CHOOSE-ROLE","operations":"checkout-oncall","comms":"CHOOSE-ROLE","first_mitigation":"CHOOSE-ACTION","verify":"CHOOSE-USER-SIGNAL"}
EOF
python3 - <<'PY2'
import json
x=json.load(open('/tmp/incident-response.json'))
placeholders=[k for k,v in x.items() if isinstance(v,str) and v.startswith('CHOOSE-')]
print('replace_these_fields',placeholders)
PY2
STEP 3

Record a timeline

Replace every CHOOSE-* value using the severity matrix and incident evidence. Record at least five timestamped entries: impact start, detection, declaration, mitigation decision/completion and SLI recovery; include one stakeholder update.

Verify

  • The severity is defended from impact/scope/urgency rather than technical drama.
  • Roles, mitigation, communications and recovery criteria are explicit; recovery uses the affected checkout SLI.

Expected outcome

  • A severity decision, explicit roles, mitigation and a minimal incident timeline.

Diagnose / troubleshoot

  • If every engineer is “incident commander,” assign one coordination owner.
  • If recovery is declared after a deploy command only, verify with black-box or service-level evidence.
Real-world connection

Structured incident response turns many parallel actions into a coherent effort to reduce user harm.

Real-tool practicum · optional but recommended

Optional practicum — inject and recover a demo failure

Use the local SRE demo service to create errors, observe Prometheus, then restore normal traffic.

Prerequisites

  • Docker/Compose available.
  • SRE Lab Pack or release source tree.

Resources created

  • Local demo service/Prometheus evidence for a controlled incident drill.

Cost / risk

  • Local CPU/RAM only; deliberate failures are generated in the demo.
TOOL 1
./ops/setup-sre-local-lab.sh
TOOL 2
for i in $(seq 1 20); do curl -s -o /dev/null http://127.0.0.1:8000/fail || true; done
TOOL 3
curl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=sre_demo:error_ratio:1m'
TOOL 4
for i in $(seq 1 50); do curl -s -o /dev/null http://127.0.0.1:8000/ok; done

Verify

  • The error signal rises after failed traffic and begins recovering after healthy traffic.

Cleanup

  • ./ops/cleanup-sre-local-lab.sh

Warnings

  • Inject failure only into the disposable demo service.
  • Mitigate/verify user impact before spending time proving root cause.
Avoid these traps

Common mistakes

  • Waiting for complete root-cause certainty before a safe mitigation.
  • Using technical component failure alone as severity.
  • Declaring recovery without checking the affected user journey.
Knowledge check

Can you connect the reliability objective to an engineering decision?

Three concept checks plus one scenario check.

1What should incident severity primarily represent?

The impact, scope and urgency of harm to users or the business.

2Why use an incident commander?

To coordinate priorities, roles and communication so responders can focus on technical work without competing command paths.

3What is the difference between mitigation and root-cause analysis?

Mitigation reduces current impact; root-cause analysis explains contributing conditions and usually continues after service is stable.

4Scenario: a bad release causes 35% checkout errors and rollback is known safe. What should the response optimize for first?

Rapidly reduce user impact by rolling back and verifying the checkout SLI, while preserving evidence for later causal analysis.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.