Gnu Group Learning AcademyGnu Group ↗
SRE / Reliability Engineering Fundamentals · Module 09 of 12

Capacity, performance and overload

Forecast demand, preserve headroom, recognize saturation early and design overload behaviour so a system degrades intentionally rather than collapsing unpredictably.

Running case · checkout service

Carry the same user journey forward: customer → edge/API → checkout → payment. Reuse prior SLI/SLO, alert, incident and recovery decisions where relevant so this module adds evidence to one reliability system rather than starting from zero.

Learning objectives

By the end of this module, you should be able to:

  • Estimate capacity headroom from demand and tested service limits.
  • Use saturation and latency trends as early warning signals.
  • Choose overload controls such as admission control, shedding, queues and graceful degradation.

Capacity is demand plus safety margin

Capacity planning combines demand forecasts, tested service limits, growth assumptions and recovery/failover requirements. A system that can barely handle today’s peak has no room for variance, maintenance or dependency degradation.

Measure the bottleneck resource or service-level limit rather than assuming CPU is always the capacity constraint.

Performance changes before hard failure

Latency often rises as a system approaches saturation. Queue depth, thread pools, connection pools, disk space or rate limits can be more informative than average CPU.

Track leading indicators and capacity exhaustion time where meaningful.

Overload should be controlled

When demand exceeds safe capacity, admission control, rate limiting, load shedding, bounded queues and degraded features can protect core transactions.

Retries without budgets or backoff can amplify overload by multiplying work.

Failover changes the headroom calculation

A service that is safe only when every instance is healthy has no maintenance or failure margin. Capacity planning should include the intended failure mode, such as losing one instance or one zone.

Example: with four equal workers, a design that must survive one worker loss should keep normal peak demand within the tested capacity of the remaining three, not the nominal four-worker total.

Choose overload controls by work semantics

Use admission control or shedding when excess work must be rejected quickly to preserve core capacity; use bounded queues when short bursts are tolerable and waiting work remains valuable; use graceful degradation when a cheaper response preserves a critical journey.

Treat 429/503 responses from otherwise valid requests as explicit degraded user outcomes and reconcile them with the SLI/error-budget policy.

Worked example

Headroom calculation

Tested sustainable capacity: 1,500 rps
Expected peak: 1,050 rps
Headroom = 450 rps = 30% of tested capacity
  • The acceptable margin depends on growth, failover and uncertainty.
  • A second dependency can still become the real bottleneck, so validate end-to-end.
Core lab · account-free

Lab 09 — Forecast saturation and headroom

Goal: Project demand growth and determine when a known capacity limit is crossed.

Before you start

  • No cloud account required.
  • Core lab uses bash/Python 3 and local files only.
  • Running case study: use the same checkout journey throughout the course unless the lab explicitly says otherwise.
STEP 1

Create weekly peaks

cat > /tmp/capacity.csv <<'EOF'
week,peak_rps
1,760
2,805
3,850
4,900
EOF
STEP 2

Estimate simple growth

python3 - <<'PY2'
import csv
r=list(csv.DictReader(open('/tmp/capacity.csv'))); vals=[int(x['peak_rps']) for x in r]
growth=(vals[-1]-vals[0])/(len(vals)-1)
limit=1200
weeks=(limit-vals[-1])/growth
print('avg_weekly_growth',round(growth,1));print('weeks_to_limit',round(weeks,1));print('current_headroom_pct',round((limit-vals[-1])*100/limit,1))
PY2
STEP 3

Define an action threshold

Choose a headroom threshold that triggers load testing, scaling or demand-control work before the hard limit.

STEP 4

Model N+1 and choose an overload response

python3 - <<'PY2'
workers=4
per_worker_safe_rps=300
peak=900
n1_capacity=(workers-1)*per_worker_safe_rps
print('nominal_capacity',workers*per_worker_safe_rps)
print('n_plus_1_capacity',n1_capacity)
print('n_plus_1_headroom',n1_capacity-peak)
for queue_ms in (20,80,250,700):
    decision='accept' if queue_ms < 100 else ('degrade/shed' if queue_ms < 500 else 'shed-fast')
    print(queue_ms,decision)
PY2

Verify

  • The calculation distinguishes tested capacity from current demand.
  • The action threshold occurs before the predicted hard limit.
  • The learner evaluates headroom under the intended failure mode and chooses queue/degrade/shed behaviour from work semantics rather than one universal threshold.

Expected outcome

  • A simple forecast, current headroom and a proactive capacity action.

Diagnose / troubleshoot

  • Linear forecasts are only approximations; revisit them when demand pattern changes.
  • If retries are enabled during overload, include their extra demand in the model.
Real-world connection

Capacity engineering creates time to act before saturation becomes an incident.

Real-tool practicum · optional but recommended

Optional practicum — inspect saturation under local load

Generate concurrent slow requests against the demo service and observe request/latency behaviour.

Prerequisites

  • Docker/Compose available.
  • SRE Lab Pack or release source tree.

Resources created

  • Local demo traffic, latency histogram and synthetic shed responses.

Cost / risk

  • Local CPU/RAM only; concurrent requests can briefly increase workstation load.
TOOL 1
./ops/setup-sre-local-lab.sh
TOOL 2
for i in $(seq 1 20); do curl -s http://127.0.0.1:8000/slow >/dev/null & done; wait
TOOL 3
for i in $(seq 1 5); do curl -s -o /dev/null http://127.0.0.1:8000/shed || true; done
TOOL 4
curl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=histogram_quantile(0.95, sum by (le) (rate(sre_demo_request_duration_seconds_bucket{outcome=~"ok|slow"}[1m])))'
TOOL 5
curl -fsS http://127.0.0.1:9090/api/v1/query --get --data-urlencode 'query=sre_demo:error_ratio:1m'

Verify

  • The learner observes both tail-latency evidence and the availability impact of shed requests.
  • The practicum explicitly connects overload protection to the SLI rather than treating 429/503 as invisible.

Cleanup

  • ./ops/cleanup-sre-local-lab.sh

Warnings

  • Keep load generation on the disposable demo service.
  • Treat shed 429/503 responses as explicit user outcomes in the SLI policy.
Avoid these traps

Common mistakes

  • Planning capacity from average traffic only.
  • Assuming CPU is the bottleneck without evidence.
  • Using unbounded retries or queues during overload.
Knowledge check

Can you connect the reliability objective to an engineering decision?

Three concept checks plus one scenario check.

1What is capacity headroom?

The margin between current or expected demand and the tested safe capacity limit.

2Why can latency be an early saturation signal?

Queues and resource contention often increase response time before a resource reaches a visible hard limit.

3What is load shedding?

Intentionally rejecting lower-priority or excess work to preserve the service for higher-priority/core requests.

4Scenario: a service is at 75% of tested peak capacity and traffic grows 10% monthly. What should SRE do?

Forecast when headroom becomes unsafe and start scaling, efficiency or demand-control work before the limit is reached.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.