Gnu Group Learning AcademyGnu Group ↗
LLMOps & AI Platform Engineering · Module 09 of 16

Reliable AI Services

AI calls are expensive and can be slow, so reliability controls must be deliberate. Classify failures, give requests a deadline, retry only failures that may recover, add backoff and jitter, stop sending work to persistently failing dependencies, and define safe degraded behavior before an incident.

Engineering focus: Timeout budgets, retries, backoff, circuit breaking and degraded operation

In plain English

What problem are we solving?

AI calls are expensive and can be slow, so reliability controls must be deliberate. Classify failures, give requests a deadline, retry only failures that may recover, add backoff and jitter, stop sending work to persistently failing dependencies, and define safe degraded behavior before an incident.

Do not memorize product names first. Build the mental model, watch the system move, then map tools onto it.

Animated mental model

Watch a transient failure either recover safely or amplify into an outage

Bounded retries can absorb brief faults; unbounded retries add load to an already unhealthy dependency.

● Loading interaction engine…
● PAUSEDCall starts with a deadline
01AI calldeadlineNOW
02Classify failurewhy failed?
03Retry policybounded
04Circuitcontain failure
05Degrade / fallbackapproved path
06Evidenceattempts + outcome
STEP 1/6Call starts with a deadline

The operation gets one finite user time budget. Every attempt consumes from the same budget.

deadline=10 s
Learning objectives

By the end of this module, you should be able to:

  • Classify transient, permanent, policy and capacity failures.
  • Design one end-to-end request deadline instead of independent fresh timeouts at every hop.
  • Apply bounded retries with backoff/jitter only to eligible failures.
  • Use circuit breaking and degraded modes to contain persistent dependency failure.
🎓 Learning Coach follows this block when it is centered in your browser

Different failure classes require different responses

A network timeout may recover on another attempt. Invalid input will not. A policy denial should not be bypassed. A GPU OOM caused by deterministic capacity pressure usually needs capacity/admission change rather than immediate retry.

Failure classification is therefore the first reliability decision. Without it, generic retry middleware can multiply cost and load while hiding the actual incident.

  • Transient: timeout, connection reset, some 5xx/provider faults.
  • Permanent request: invalid schema, unsupported model/parameter.
  • Policy: authentication/authorization/residency denial.
  • Capacity: queue full, admission rejection, deterministic memory pressure.
Common beginner misconception

Do not turn every exception into 'provider unavailable'. That destroys the evidence needed to choose a correct recovery action.

🎓 Learning Coach follows this block when it is centered in your browser

A request needs one finite time budget across all attempts

If every layer starts a fresh 30-second timeout, a user request can take minutes as gateway, backend and fallback each consume a full timer. A deadline budget solves this by measuring how much time remains for the whole operation.

Retries and fallbacks consume that remaining budget. When too little time remains for a useful attempt, fail clearly instead of starting work that cannot satisfy the user objective.

  • Client sets or platform derives an end-to-end deadline.
  • Gateway/backend propagate remaining time.
  • Retry/fallback does not reset the original budget.
  • Cancellation should stop expensive work when the caller no longer needs it.
Common beginner misconception

A timeout value at one service boundary is not the same as an end-to-end user deadline.

🎓 Learning Coach follows this block when it is centered in your browser

Retries need bounds; circuit breakers stop feeding a failing dependency

A small number of retries can hide brief faults. Backoff spaces attempts so recovery has time to occur, and jitter prevents many clients from retrying in lockstep.

When failure persists, a circuit breaker temporarily stops normal calls to the unhealthy dependency. This protects both the dependency and callers from spending resources on work that is very unlikely to succeed.

  • Retry only eligible failure classes.
  • Set a maximum attempt count or retry budget.
  • Use backoff and jitter for shared dependencies.
  • Open a circuit on sustained failure; probe recovery deliberately.
Common beginner misconception

Circuit breakers are not only about errors; slow timeouts can be just as damaging because they consume connection and concurrency capacity.

🎓 Learning Coach follows this block when it is centered in your browser

Reliability includes deciding what acceptable degraded service looks like

Not every AI feature deserves the same failure behavior. A chat assistant may switch to a smaller approved model, a summarizer may queue work for later, and a high-risk decision system may need to fail closed rather than use a weaker model.

Degraded modes should be designed and tested before incidents. They need clear user signaling, policy constraints and a way to return to normal operation.

  • Smaller/local model fallback where quality/security permit.
  • Read-only/cached response or deferred processing for suitable workflows.
  • Disable optional AI feature while keeping core product available.
  • Fail closed for high-risk tasks where degraded quality is unacceptable.
Common beginner misconception

Fallback success is not the same as full service health; user experience or quality may be reduced.

Interactive simulator

Failure injection simulator: retry, fallback or fail fast?

Inject timeout, rate-limit, policy and capacity failures. Tune retry count and watch whether the system recovers or creates extra load.

TEACHING MODEL
● Loading simulator engine…
Attempts—
Circuit—
Outcome—
Risk—

Choose a failure class, then inject it.

This simulator teaches relationships and diagnostic reasoning. Its synthetic values are not vendor benchmarks, hardware-sizing evidence or production SLO measurements.

Worked examples

Retry eligibility table

failure                 action
timeout/connection reset  bounded retry + backoff
429 rate limit             honor backoff/retry guidance
invalid request            do not retry
authorization denied       do not retry/bypass
deterministic GPU OOM      capacity/admission fix
  • The same 'request failed' symptom maps to different actions.
  • Retries are chosen from failure semantics, not convenience.

One deadline across fallback

user deadline: 10s
primary: 3s timeout
backoff: 0.5s
retry: 2s failure
remaining: 4.5s
fallback typical: 3s -> allowed

Do not reset deadline to another 10s.
  • Retries consume the original user budget.
  • The fallback decision considers remaining time.
Try it · core lab · account-free

Inject failures and compare bounded recovery with a retry storm

Goal: See why error classification, retry budgets and circuit breaking are reliability controls rather than generic exception handling.

Before you start

  • Python 3 only. The dependency is a deterministic simulated model provider.
STEP 1

Recover a transient failure

python3 public/labs/llmops/09_reliability_controls.py --scenario transient
STEP 2

Reject a policy failure

python3 public/labs/llmops/09_reliability_controls.py --scenario policy
STEP 3

Open the circuit

python3 public/labs/llmops/09_reliability_controls.py --scenario persistent
STEP 4

Compare with unsafe retry storm

python3 public/labs/llmops/09_reliability_controls.py --scenario storm
Download this Python labDownload all current labs

Verify

  • Transient scenario succeeds within a bounded attempt count.
  • Policy scenario performs exactly one denied attempt.
  • Persistent scenario opens the circuit and stops feeding repeated work.
  • Storm scenario demonstrates amplified attempt count and is marked unsafe.

Expected outcome

  • You can map failure class to retry/fallback/reject behavior.
  • You can explain why reliability must bound both time and attempt count.

Break / fix · diagnose

  • If policy errors retry, inspect the lab's retryable error set.
  • If persistent failure never opens the circuit, inspect the consecutive-failure threshold.
Production view

Use shared retry/deadline policy across gateway and inference clients so nested libraries do not multiply attempts during provider or capacity incidents.

Production Engineer checklist

Before you call this concept production-ready:

This checklist is a study aid, not a proficiency score. Your evidence record remains based on tracked learning activity and later formal assessment.

Avoid these traps

Common mistakes

  • Retrying every non-200 response.
  • Starting a fresh full timeout for every retry/fallback layer.
  • Using fallback to bypass a hard policy or quality requirement.
  • Hiding sustained primary failure because fallback still returns responses.
Knowledge check

Reason about the system, not just the vocabulary.

Answer before opening each explanation. Opening all four answers records knowledge-check activity in My Learning.

1Why should an invalid request normally not be retried unchanged?

The failure is permanent for that request; repeating it adds load/cost without changing the invalid input.

2What is the purpose of an end-to-end deadline budget?

Bound total user wait and prevent multiple layers/attempts from each consuming a fresh full timeout.

3What does a circuit breaker protect during persistent failure?

It stops repeatedly sending work to a dependency that is very likely to fail, reducing wasted resources and cascading pressure.

4What makes a degraded mode production-ready?

It is predesigned, policy-compliant, observable, tested and clear about reduced capability/quality.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.