SRE / Reliability Engineering Fundamentals · Module 03 of 12

SLOs and error budgets

Set user-focused reliability targets, calculate the allowable bad-event budget exactly, define an error-budget policy and use the result to guide engineering decisions rather than as a ceremonial percentage.

Running case · checkout service

Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.

Learning objectives

By the end of this module, you should be able to:

Distinguish SLI, SLO and SLA without conflating engineering objectives with contractual commitments.
Calculate error budgets for event-based SLOs without floating-point ambiguity.
Write an error-budget policy that defines stakeholders, triggers and actions.

SLOs put a target on an SLI

An SLO states the desired level of a service indicator over a defined window. The target should be driven by user needs and business consequences, not by the number of nines that sounds impressive.

A useful SLO names the SLI, target, window, exclusions, data source and ownership.

The error budget is the allowed unreliability

For an event-based SLO, the error budget is the fraction of valid events allowed to be bad: 1 minus the target. For 99.9%, the budget is 0.1% of valid events.

Use integer event counts when possible so rounding does not create accidental policy ambiguity.

Policies turn measurement into action

An SLO does not improve reliability by itself. A policy defines what happens when the budget is healthy, at risk or exhausted, who participates in the decision and how exceptions are recorded.

Treat the policy as a coordination mechanism across product, engineering and reliability stakeholders rather than a punishment mechanism.

SLOs and SLAs serve different purposes

An SLO is an engineering/product objective for service behaviour. An SLA is an agreement that may include contractual commitments or consequences when a defined service level is missed.

A team can choose an internal SLO that is stricter than an external SLA so it has room to detect and correct reliability erosion before a contractual boundary is threatened.

Window semantics change the decision

A rolling window continuously evaluates the most recent period and is useful for operational control. A calendar window resets on a fixed boundary and may align better with reporting or commercial commitments.

Document the window, exclusions and late-arriving data behaviour. The same target can produce different decisions if the measurement window is ambiguous.

Worked example

Exact event budget

Target: 99.9%
Valid events in window: 2,000,000
Allowed bad events = floor(2,000,000 × 1 / 1000) = 2,000

Represent 99.9% as 999/1000 to avoid binary floating-point surprises.
The same objective can be expressed with a good-event threshold of 1,998,000.

Worked error-budget policy

GREEN: <50% budget used at mid-window → normal change policy
AT RISK: fast burn or >75% used → reliability review + reduce risky change
EXHAUSTED: budget spent → freeze discretionary risk; allow fixes/security/emergency changes with explicit owner
RECOVERY: resume normal change after objective/policy recovery criteria are met

Thresholds are an example policy, not a universal standard.
The important property is that owners, triggers, allowed exceptions and recovery conditions are explicit before an incident.

Core lab · account-free

Lab 03 — Build an exact SLO budget calculator

Goal: Calculate good-event requirements, bad-event budget and remaining budget using integer arithmetic.

Before you start

No cloud account required.
Core lab uses bash/Python 3 and local files only.
Running case study: use the same checkout journey throughout the course unless the lab explicitly says otherwise.

STEP 1

Calculate the target as a rational number

python3 - <<'PY2'
from fractions import Fraction
total=2_000_000
target=Fraction(999,1000)
exact_bad=(total*(target.denominator-target.numerator))//target.denominator
print('allowed_bad',exact_bad)
print('required_good',total-exact_bad)
PY2

STEP 2

Add observed failures

python3 - <<'PY2'
total=2_000_000; allowed_bad=2_000; observed_bad=650
print('remaining_bad',allowed_bad-observed_bad)
print('budget_used_pct',f'{observed_bad*100/allowed_bad:.1f}%')
PY2

STEP 3

Write a three-state policy

Define a normal state, at-risk state and exhausted state with concrete engineering actions and named decision owners.

Verify

The calculator returns exactly 2,000 allowed bad events.
The policy changes behaviour based on reliability evidence and has accountable owners.

Expected outcome

An exact error-budget calculation and a decision policy that can be reviewed with product/engineering stakeholders.

Diagnose / troubleshoot

Do not confuse 99.9% reliability with 99.9% of the error budget remaining.
If the policy says only “stop releases,” add investigation, mitigation, exception and recovery criteria.

Real-world connection

Error budgets make reliability trade-offs observable and governable rather than intuitive.

Real-tool practicum · optional but recommended

Optional practicum — inspect SLO burn in Prometheus

Use the local lab recording rules to query the current error ratio and derived burn rate.

Prerequisites

Docker/Compose available.
SRE Lab Pack or release source tree.

Resources created

Local demo service and Prometheus recording rules for error ratio and burn rate.

Cost / risk

Local CPU/RAM only; deliberate demo failures consume the synthetic error budget.

TOOL 1

./ops/setup-sre-local-lab.sh

TOOL 2

for i in 1 2 3; do curl -s -o /dev/null http://127.0.0.1:8000/fail || true; done

TOOL 3

curl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=sre_demo:burn_rate:1m'

Verify

Prometheus exposes a burn-rate result relative to the demo 99.9% objective.

Cleanup

./ops/cleanup-sre-local-lab.sh

Warnings

Inject failures only into the disposable demo service.
A burn-rate value is meaningful only relative to a defined SLO/error budget.

Avoid these traps

Common mistakes

Choosing an SLO solely because a competitor advertises the same number.
Treating an SLA and an internal SLO as interchangeable.
Publishing an error budget with no agreed decision policy.

Knowledge check

Can you connect the reliability objective to an engineering decision?

Three concept checks plus one scenario check.

1How is an error budget related to an SLO?

It is the allowed unreliability: for a target expressed as a fraction, error budget equals 1 minus the SLO target.

2What makes an SLO actionable?

A defined SLI, target, window, ownership and a policy that maps budget state to engineering decisions.

3Why use integer or rational arithmetic in event-budget calculations?

It avoids floating-point rounding ambiguity when policy depends on exact allowed bad-event counts.

4Scenario: a team is within its SLO but has consumed 90% of the monthly error budget in the first week. What should happen?

Treat the budget as at risk, investigate the burn pattern and apply the agreed policy before the service exhausts its reliability allowance.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.