Gnu Group Learning AcademyGnu Group ↗
SRE / Reliability Engineering Fundamentals · Module 11 of 12

Postmortems, toil and reliability improvement

Turn incidents into system learning through blameless analysis, evidence-backed action items, toil reduction and a reliability roadmap that actually changes future behaviour.

Running case · checkout service

Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.

Learning objectives

By the end of this module, you should be able to:

  • Write a blameless postmortem focused on system conditions and decision context.
  • Prioritize corrective actions by risk reduction, evidence and ownership.
  • Connect incident learning and toil measurements to an ongoing reliability roadmap.

Blameless does not mean accountability-free

A blameless postmortem avoids treating individual error as the endpoint of analysis. It reconstructs the context, safeguards, incentives and system conditions that made the outcome possible.

Actions still need owners, due dates and verification. The objective is better system behaviour, not softer language.

Action items must change future risk

“Be more careful” and “remind the team” are weak corrective actions. Prefer changes to tests, rollout controls, observability, isolation, capacity, automation or decision processes that can be verified.

Rank actions by expected risk reduction and effort; not every observation needs a project.

Reliability improvement is continuous

Aggregate recurring incident themes, page sources and toil patterns to find systemic investments. Reliability work competes for engineering capacity and should be justified with evidence just like feature work.

Track whether action items actually close and whether recurrence, detection time or impact improves.

A useful postmortem has a stable structure

Capture impact, timeline, detection, response, contributing conditions, what went well/poorly, corrective actions, owners and verification. Avoid a single-root-cause story when several conditions combined to produce the incident.

Classify follow-up as prevention, detection, mitigation, recovery, documentation/training or accepted risk so the portfolio is not dominated by one kind of action.

Worked example

Action quality

Weak: remind engineers to check the flag
Stronger: add pre-deploy validation that rejects incompatible flag state and test rollback
  • The stronger action changes the system and can be verified.
  • Human training can complement, but should not substitute for a missing technical guardrail.

Blameful → systems-focused rewrite

Blameful: The engineer forgot to enable the safety flag.
Systems-focused: The deployment path allowed an incompatible flag state, the preflight did not validate it, and rollback guidance did not surface the dependency.
  • The rewrite preserves accountability for improvement without treating one person as the root cause.
  • Strong follow-up changes the deployment control and verifies the new safeguard.
Core lab · account-free

Lab 11 — Grade postmortem actions and toil ROI

Goal: Score corrective actions for durability and estimate time recovered by one automation.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
  • Running case study: use the same checkout journey throughout the course unless the lab explicitly says otherwise.
STEP 1

Classify candidate postmortem actions

python3 - <<'PY2'
actions=[
 ('remind team to check flag','training'),
 ('reject incompatible flag during preflight','prevention'),
 ('page on checkout fast burn','detection'),
 ('automate signed rollback after approved trigger','mitigation')
]
for action,category in actions:
    print(f'{category:10} | {action}')
print('\nNow add one recovery action and one accepted-risk item of your own.')
PY2
STEP 2

Compute response intervals from impact start

cat > /tmp/timeline.json <<'EOF'
[{"event":"impact_start","minute":0},{"event":"detect","minute":4},{"event":"declare","minute":7},{"event":"mitigate","minute":20},{"event":"recover","minute":31}]
EOF
python3 - <<'PY2'
import json
r={x['event']:x['minute'] for x in json.load(open('/tmp/timeline.json'))}
print('TTD_minutes',r['detect']-r['impact_start'])
print('TTM_minutes',r['mitigate']-r['impact_start'])
print('TTR_minutes',r['recover']-r['impact_start'])
PY2
STEP 3

Write a follow-up contract

Rewrite one blameful statement into a systems-focused observation. Then choose one follow-up action, name its owner/due date, classify it, and define evidence that proves the action reduced detection, mitigation, recurrence or toil.

Verify

  • TTD, time-to-mitigate and time-to-recover are calculated from impact start rather than from detection.
  • The postmortem rewrite focuses on system conditions while follow-up actions still have accountable owners and verification.

Expected outcome

  • One durable corrective action and one measurable reliability-improvement investment.

Diagnose / troubleshoot

  • A postmortem with many ownerless actions is not a learning system.
  • Do not automate toil when the underlying operation should be deleted instead.
Real-world connection

Postmortems and toil data are inputs to engineering prioritization, not archival documents.

Real-tool practicum · optional but recommended

Optional practicum — turn an incident timeline into structured evidence

Use a local JSON timeline and jq/Python to summarize detection, mitigation and recovery intervals.

Prerequisites

  • Python 3.

Resources created

  • A temporary incident timeline JSON under /tmp.

Cost / risk

  • No cloud cost.
TOOL 1
cat > /tmp/timeline.json <<'EOF'
[{"event":"impact_start","minute":0},{"event":"detect","minute":4},{"event":"declare","minute":7},{"event":"mitigate","minute":20},{"event":"recover","minute":31}]
EOF
TOOL 2
python3 - <<'PY2'
import json
r={x['event']:x['minute'] for x in json.load(open('/tmp/timeline.json'))}
for name,end in [('TTD','detect'),('TTM','mitigate'),('TTR','recover')]:
    print(name,r[end]-r['impact_start'])
PY2

Verify

  • The timeline contains an impact-start event and produces TTD, TTM and TTR from the same origin.

Cleanup

  • Remove only resources created for the disposable lab.

Warnings

  • Use synthetic or sanitized incident data.
  • Do not use TTD/TTM/TTR as individual-performance scores; they are system/process learning signals.
Avoid these traps

Common mistakes

  • Ending a postmortem at “human error.”
  • Creating action items with no owner or verification.
  • Tracking toil without funding any engineering work to reduce it.
Knowledge check

Can you connect the reliability objective to an engineering decision?

Three concept checks plus one scenario check.

1What does “blameless” mean in a postmortem?

Analyze system conditions and decision context without making individual blame the endpoint, while still assigning accountable improvement work.

2What makes a corrective action strong?

It changes a control, system or process in a verifiable way that reduces recurrence, impact or response cost.

3How can toil data influence a reliability roadmap?

It quantifies recurring operational cost and helps prioritize engineering changes that recover time or reduce risk.

4Scenario: a postmortem action says “engineers should remember to check the flag next time.” How should it be improved?

Add a technical or procedural guardrail such as pre-deploy validation that detects the incompatible state and verifies rollback behaviour.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.