Design alerts that page on actionable threats to reliability objectives, calculate burn rate and use multiple windows to distinguish fast outages from slower budget erosion.
Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.
Paging is expensive and should be reserved for conditions where a human can improve the outcome now. Alerts that are not urgent belong in tickets, reports or dashboards.
Tie the trigger to an SLO or another explicit user-impact/risk condition whenever practical.
Burn rate compares the observed bad-event rate with the error rate allowed by the SLO. A burn rate of 1 consumes the budget exactly across the full window; higher values spend it faster.
For a 99.9% objective, the allowed bad-event rate is 0.1%. An observed 1.44% bad-event rate is a 14.4× burn.
A long window confirms material budget consumption while a short window confirms the problem is still active. This reduces stale pages and helps distinguish fast severe outages from slower reliability erosion.
Use published starting points as guidance, then tune them to traffic volume, risk and sustainable pager load.
For a full SLO period T, consuming fraction p of the full error budget inside a shorter window w corresponds to burn rate p × T / w when T and w use the same units.
For a 30-day (720-hour) objective: 2% in 1 hour gives 14.4×; 5% in 6 hours gives 6×; 10% in 72 hours gives 1×. These are useful starting points, not immutable constants.
SLO = 99.9% → allowed error rate = 0.1%
Observed error rate = 1.44%
Burn rate = 1.44 / 0.10 = 14.4×Goal: Calculate burn rates and map them to page, ticket or observe decisions.
python3 - <<'PY2'
period_hours=30*24
cases=[('page-fast',0.02,1),('page-slow',0.05,6),('ticket',0.10,72)]
for name,budget_fraction,window_hours in cases:
burn=budget_fraction*period_hours/window_hours
print(name,'burn_rate',burn)
PY2python3 - <<'PY2'
slo=0.999
valid=50_000
bad=720
allowed_error=1-slo
observed_error=bad/valid
print('observed_error_pct',round(observed_error*100,3))
print('burn_rate',round(observed_error/allowed_error,1))
print('decision','PAGE' if observed_error/allowed_error >= 14.4 else 'evaluate slower windows')
PY2Now change `bad` to 50, 300 and 1,500. Explain why a single short-window ratio is not enough to determine urgency, especially at low traffic.
SLO-based alerting protects the error budget while controlling pager noise.
Use promtool inside the local Prometheus container and query the recording rules.
./ops/setup-sre-local-lab.shdocker exec gnu-sre-prometheus promtool check rules /etc/prometheus/slo-alerts.ymlcurl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=sre_demo:burn_rate:1h'curl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=sre_demo:burn_rate:5m'Three concept checks plus one scenario check.
The service is consuming error budget at exactly the rate allowed by the SLO across the full objective window.
The long window confirms meaningful budget consumption while the short window verifies the problem is still active.
When it represents urgent, actionable risk or user impact where human intervention can improve the outcome now.
Evaluate low-traffic handling such as longer windows, combining signals or synthetic traffic so isolated events do not create unsustainable pages.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.