Escalation contract
0–5 min: primary on-call
5–10 min: secondary
SEV-1 or no mitigation path: incident commander + service owner- Time bounds remove ambiguity.
- Escalation is based on service need, not individual heroics.
SRE / Reliability Engineering Fundamentals · Module 06 of 12
On-call, escalation and handoffs
Design sustainable on-call coverage with actionable pages, explicit escalation, handoff discipline, responder safety and enough context for the person receiving the page to act.
Running case · checkout service
Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.
Learning objectives
By the end of this module, you should be able to:
- Define the responsibilities and limits of an on-call engineer.
- Design an escalation policy with time bounds and fallback ownership.
- Evaluate pager load and handoff quality as reliability-system inputs.
On-call is part of the production design
A service is not operationally ready if the alerting system can detect failure but nobody is clearly responsible for responding. On-call ownership, access, runbooks and escalation are production dependencies.
Design the system so the primary responder can identify impact, mitigate safely and obtain help without relying on tribal knowledge.
Escalation must be explicit
An escalation policy should answer who is primary, how long to wait, who is secondary, when incident command is activated and how cross-team dependencies are engaged.
Avoid escalation chains that depend on personal phone books or informal knowledge.
Sustainable pager load is an engineering signal
Repeated non-actionable pages, excessive overnight interruptions or pages caused by known toil indicate system work is needed. Do not normalize responder exhaustion as a staffing problem alone.
Measure page volume, actionability, time-to-acknowledge and common page sources, then improve the underlying system.
Measure the on-call system, not just the people
Useful operating measures include pages per shift, percentage of actionable pages, acknowledgement latency, escalations, overnight interruptions and recurring alert sources. Set team-specific load budgets rather than copying a universal number.
Rotation size also matters: a very small rotation increases frequency and fatigue, while a large rotation can reduce familiarity. Balance sustainable frequency with enough production context to respond safely.
Worked example
Worked pager-load review
7 engineers · weekly primary rotation
14 pages/week → 2 pages/day average
10 actionable, 4 non-actionable → 71% actionability
Decision: fix the 4 noisy alerts before treating staffing as the primary solution- These numbers are an example for reasoning, not a universal SRE target.
- Trend page volume and actionability by service and time-of-day so the engineering cause of load remains visible.
Core lab · account-free
Lab 06 — Validate an on-call policy
Goal: Check a small policy for missing ownership, time bounds and escalation coverage.
Before you start
- No cloud account required.
- Core lab uses bash/Python 3 and local files only.
- Running case study: use the same checkout journey throughout the course unless the lab explicitly says otherwise.
Start with an incomplete policy
cat > /tmp/oncall.json <<'EOF'
{"primary":"checkout-oncall","secondary":"","ack_minutes":10,"escalate_minutes":5,"sev1_commander":"","handoff_required":false,"pages_last_7d":14,"actionable_pages":10}
EOFRun a validator and inspect every failure
python3 - <<'PY2'
import json
x=json.load(open('/tmp/oncall.json'))
issues=[]
for k in ['primary','secondary','sev1_commander']:
if not x.get(k): issues.append(f'missing {k}')
if not (0 < x.get('ack_minutes',0) < x.get('escalate_minutes',0)): issues.append('ack/escalation time order is invalid')
if x.get('handoff_required') is not True: issues.append('handoff is not mandatory')
pages=x.get('pages_last_7d',0); actionable=x.get('actionable_pages',0)
print('actionability_pct',round(actionable*100/pages,1) if pages else 0)
print('issues',issues)
assert issues, 'This starting policy is intentionally broken'
PY2Write a handoff minimum
Edit `/tmp/oncall.json` until it has durable primary/secondary roles, ack < escalation time, an incident-command path and mandatory handoff. Then rerun the validator after changing the final assertion to `assert not issues, issues`. State what you would do about the four non-actionable pages.
Verify
- The repaired policy has primary, secondary, time-bound escalation, incident-command coverage and mandatory handoff.
- The learner calculates actionability and proposes engineering work for noisy pages rather than treating fatigue as a staffing-only problem.
Expected outcome
- A machine-checkable policy skeleton and human-readable handoff contract.
Diagnose / troubleshoot
- If escalation names individual people, prefer durable role/group identifiers.
- If pages regularly require permissions the on-call role lacks, fix the access model before calling the service ready.
Real-world connection
Reliable services include a reliable human-response system.
Real-tool practicum · optional but recommended
Optional practicum — check Alertmanager routing syntax
Use the existing/local Prometheus tool container to lint a disposable Alertmanager-style routing file if amtool is available.
Prerequisites
- A shell; `amtool` is optional for linting.
Resources created
- A temporary Alertmanager-style YAML file under /tmp.
Cost / risk
- No cloud cost; no notification is sent by this practicum.
printf '%s\n' 'route:' ' receiver: primary-oncall' 'receivers:' '- name: primary-oncall' > /tmp/alertmanager.ymlcommand -v amtool >/dev/null && amtool check-config /tmp/alertmanager.yml || echo 'amtool not installed; review syntax manually'Verify
- The routing file has an explicit default receiver and no secret material.
Cleanup
- Remove only resources created for the disposable lab.
Warnings
- Do not place real phone numbers, webhook secrets or tokens in the sample routing file.
- Syntax validation does not prove the rotation is sustainable or the escalation policy is complete.
Avoid these traps
Common mistakes
- Treating on-call as an informal favour.
- Allowing pages that provide no user impact or first diagnostic step.
- Using individual names instead of durable escalation roles.
Knowledge check
Can you connect the reliability objective to an engineering decision?
Three concept checks plus one scenario check.
1What makes an on-call escalation policy operationally useful?
It defines durable owners, time bounds, fallback paths and conditions for broader incident coordination.
2Why is pager load a reliability concern?
Excessive or non-actionable pages degrade responder performance and reveal alerting or system-design work that should be fixed.
3What should a shift handoff preserve?
Active operational context such as incidents, mitigations, risky changes, suppressed alerts and pending actions.
4Scenario: the primary on-call is paged but lacks permission to perform the documented mitigation. What failed?
Operational readiness: the on-call role, access model and runbook were not aligned with the actions required to restore service.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.