Design systems and delivery practices that contain failure: timeouts, bounded retries, isolation, graceful degradation, canaries, rollback and recovery objectives tied to tested evidence.
Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.
Timeouts stop requests from waiting forever; retries can recover transient failures but must be bounded, delayed and budgeted; bulkheads or isolation prevent one workload from consuming every shared resource.
Define attempts versus retries precisely. Three total attempts at each of three layers can create 27 backend attempts; three retries means four attempts at each layer and can create 64. Prefer retrying at one appropriate layer, with deadlines, backoff, jitter and a retry budget.
When a noncritical dependency fails, a service can disable recommendations, serve cached data or reduce expensive features while keeping the critical transaction available.
Document what can degrade and what must fail closed for correctness, security or compliance.
Canaries, staged rollout and automated rollback reduce blast radius when release health is evaluated with user-facing signals. A deployment system saying “success” does not prove the service is healthy.
Recovery objectives should distinguish the recovery time objective (RTO)—how quickly service must return—from the recovery point objective (RPO)—how much data loss is acceptable, then validate both with exercises rather than assuming backups or failover will meet them.
A circuit breaker temporarily stops calls to a dependency that is predictably failing, then probes recovery instead of continuing to spend resources on work likely to fail. It is a containment mechanism, not a substitute for capacity or dependency repair.
For side-effecting operations such as payments, retries require idempotency. An idempotency key or equivalent transaction contract lets a repeated request return the original outcome rather than charging the customer twice.
3 TOTAL ATTEMPTS per layer × 3 layers = 3³ = 27 backend attempts
3 RETRIES per layer means 4 attempts × 3 layers = 4³ = 64 backend attemptsGoal: Calculate retry amplification and design a bounded recovery policy.
python3 - <<'PY2'
layers=3
for retries in (2,3):
attempts=1+retries
print(f'{retries}_retries_per_layer => {attempts}^{layers} = {attempts**layers} backend attempts')
print('three_total_attempts_per_layer =>',3**layers)
PY2Design one retry layer with: total-attempt limit, end-to-end deadline, exponential backoff + jitter, a retry budget, retriable/non-retriable status policy and a circuit-breaker open/half-open rule.
For checkout/payment, define an idempotency-key rule and one graceful-degradation choice. Then set an RTO and RPO for a regional failure and state the game-day test that would prove them.
Resilience is the ability to contain failure and restore the user journey with predictable trade-offs.
Generate failures, recover with healthy traffic and compare the Prometheus error signal before/after.
./ops/setup-sre-local-lab.shfor i in $(seq 1 10); do curl -s -o /dev/null http://127.0.0.1:8000/fail || true; donecurl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=sre_demo:error_ratio:1m'for i in $(seq 1 40); do curl -s -o /dev/null http://127.0.0.1:8000/ok; doneThree concept checks plus one scenario check.
Unbounded or layered retries can amplify load, extend latency and worsen an overloaded dependency.
Preserving the critical user journey by reducing or disabling nonessential capability when dependencies or capacity are impaired.
Independent service/user health evidence and explicit rollback criteria, not deployment completion alone.
Treat the change as unhealthy and follow the rollback/mitigation policy even though the deployment system reports success.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.