SRE / Reliability Engineering Fundamentals · Module 12 of 12

Capstone: production reliability system

Assemble a defensible reliability operating model for one service: user journey, SLI/SLO, error budget, alerting, on-call, incident workflow, capacity, safe change, runbook and postmortem evidence.

Running case · checkout service

Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.

Learning objectives

By the end of this module, you should be able to:

  • Design an end-to-end reliability contract for a production-shaped service.
  • Validate that monitoring, alerting, response and recovery all reference the same user objective.
  • Defend reliability decisions with evidence instead of tool screenshots.

One objective should connect the system

The capstone is relationship-aware. The user journey, SLI, SLO, burn alert, incident severity, recovery check and postmortem must describe the same reliability concern.

A disconnected dashboard or generic runbook is not enough.

Evidence must survive disagreement

A reviewer should be able to inspect the event population, objective, observed failures, alert decision, incident timeline, rollback evidence and follow-up actions without relying on the learner’s narrative alone.

Make assumptions explicit and keep calculations reproducible.

Operational readiness includes humans and capacity

The final design needs ownership, escalation, safe access, capacity headroom, rollback, recovery objectives and corrective-action tracking. Reliability is a socio-technical system, not a monitoring feature.

The capstone is formative evidence, not a hidden certificate gate

The relationship-aware validator checks structural consistency, but it cannot judge whether the chosen SLO, severity decision or resilience trade-off is appropriate. Use the rubric for self-review, peer review or instructor review.

For this Academy release, certificate eligibility is determined by the timed assessment and payment/issuance workflow; the capstone is recommended learning evidence and is not an additional automated pass/fail requirement.

Worked example

Capstone evidence chain

Journey → SLI → SLO/error budget → burn alert → on-call → incident → mitigation → SLI recovery → postmortem → reliability backlog
  • Every arrow must be supported by a concrete relationship.
  • The validator intentionally rejects inconsistent SLO, ownership or recovery evidence.
Core lab · account-free

Lab 12 — Validate a reliability evidence bundle

Goal: Run the supplied relationship-aware validator against a production-shaped reliability record.

Before you start

  • No cloud account required.
  • Python 3 is required. You may use the release source tree or download the public SRE Lab Pack from the course landing page.
  • Running case study: use the same checkout journey throughout the course unless the lab explicitly says otherwise.
STEP 1

Get the capstone assets

if [ -f labs/sre-capstone/reliability.example.json ]; then
  cp labs/sre-capstone/reliability.example.json /tmp/reliability.json
  cp labs/sre-capstone/validate_reliability.py /tmp/validate_reliability.py
else
  curl -fsS https://learn.gnugroup.org/academy/labs/sre/reliability.example.json -o /tmp/reliability.json
  curl -fsS https://learn.gnugroup.org/academy/labs/sre/validate_reliability.py -o /tmp/validate_reliability.py
fi
STEP 2

Validate the relationships

python3 /tmp/validate_reliability.py /tmp/reliability.json
STEP 3

Break one relationship deliberately

python3 - <<'PY2'
import json
p='/tmp/reliability.json'
x=json.load(open(p))
x['incident']['recovery_sli']='wrong-sli'
json.dump(x,open('/tmp/reliability-bad.json','w'),indent=2)
PY2
python3 /tmp/validate_reliability.py /tmp/reliability-bad.json || echo 'expected: validator rejected inconsistent recovery evidence'
STEP 4

Defend the design in writing

Write a 150–250 word defense covering: why the SLI represents the user journey; why the SLO/error-budget policy is appropriate; why the alert deserves its urgency; and which rollback/recovery evidence would change your decision. The validator cannot grade this reasoning—use the rubric for self/peer/instructor review.

Verify

  • The supplied evidence bundle passes.
  • Changing the incident recovery SLI to an unrelated indicator causes validation to fail.
  • The learner can defend the reliability choices in prose; the validator alone is not treated as sufficient evidence.

Expected outcome

  • A coherent reliability contract plus a short written defense suitable for rubric-based self, peer or instructor review.

Diagnose / troubleshoot

  • If a field exists only to satisfy the validator, explain the operational decision it represents.
  • If rollback or escalation has no owner, the system is not operationally complete.
Real-world connection

The capstone demonstrates that reliability is a connected operating system of objectives, telemetry, decisions, responders and learning—not a collection of isolated practices.

Real-tool practicum · optional but recommended

Optional practicum — run an end-to-end local reliability drill

Start the demo service and Prometheus, inject failure, inspect burn, recover and save the evidence.

Prerequisites

  • Docker/Compose and Python 3.
  • SRE Lab Pack or release source tree.

Resources created

  • Local demo service, Prometheus evidence and a capstone reliability bundle.

Cost / risk

  • Local CPU/RAM only.
TOOL 1
./ops/setup-sre-local-lab.sh
TOOL 2
for i in $(seq 1 20); do curl -s -o /dev/null http://127.0.0.1:8000/fail || true; done
TOOL 3
curl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=sre_demo:burn_rate:1m' > /tmp/sre-burn.json
TOOL 4
for i in $(seq 1 50); do curl -s -o /dev/null http://127.0.0.1:8000/ok; done
TOOL 5
cp labs/sre-capstone/reliability.example.json /tmp/reliability.json && python3 labs/sre-capstone/validate_reliability.py /tmp/reliability.json

Verify

  • The learner can show failure evidence, recovery evidence and a coherent capstone contract.

Cleanup

  • ./ops/cleanup-sre-local-lab.sh

Warnings

  • Run failure injection only in the disposable lab.
  • The validator checks relationships; the written defense and rubric are still required for meaningful review.
Capstone rubric

Score connected reliability evidence—not tool polish.

User journey + SLI contract · 15%

Critical user journey, valid/good event definitions, measurement location and limitations.

SLO + error-budget policy · 15%

Target/window, exact budget calculation, stakeholders and policy actions.

Monitoring + alerting · 15%

Golden-signal context, burn-rate alert, urgency/actionability and low-traffic considerations.

On-call + incident response · 15%

Ownership, escalation, severity, roles, timeline, mitigation and communication.

Capacity + resilience · 15%

Headroom, overload controls, timeout/retry policy, graceful degradation and safe-change rollback.

Troubleshooting + runbook · 10%

Symptom-first hypotheses, discriminating checks, safe actions, verification and escalation.

Postmortem + improvement · 15%

Blameless causal analysis, accountable actions, toil/reliability roadmap and verification metrics.

Avoid these traps

Common mistakes

  • Showing tool screenshots without a user reliability objective.
  • Using one SLI for the SLO and a different unrelated signal to declare incident recovery.
  • Leaving ownership, escalation or rollback implicit.
Knowledge check

Can you connect the reliability objective to an engineering decision?

Three concept checks plus one scenario check.

1What makes the SRE capstone relationship-aware?

The user journey, SLI, SLO, alerts, incident recovery and learning evidence must refer to a coherent reliability objective.

2Why is a dashboard screenshot insufficient capstone evidence?

It does not prove the event population, objective, alert decision, response ownership, recovery or corrective-action relationships.

3What operational evidence should accompany a reliability objective?

At minimum ownership, alert/response policy, escalation, mitigation/rollback, capacity/recovery context and measurable follow-up.

4Scenario: the incident is declared recovered because CPU normalized, but the checkout SLI is still below objective. Is recovery proven?

No. Recovery must be verified against the affected user-facing service indicator or another explicitly justified recovery criterion.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.