Gnu Group Learning AcademyGnu Group ↗
AI Engineering Fundamentals · Module 11 of 12

Run AI reliably and control cost

Design fallbacks, routing, concurrency and service objectives so the AI feature can handle traffic, failures and budget limits while making explicit trade-offs between quality, latency and cost.

Engineering focus: Production architecture, economics and reliability

In plain English

What problem are we solving?

Design fallbacks, routing, concurrency and service objectives so the AI feature can handle traffic, failures and budget limits while making explicit trade-offs between quality, latency and cost.

The rest of this module introduces the formal terminology and production controls behind that idea. You are not expected to memorize the vocabulary before using it.

Running case · Atlas Assist

Carry the same support assistant forward: approved KB, source citations, explicit insufficient-evidence fallback and one bounded read-only asset lookup. Reuse prior artifacts so each module adds evidence to one production AI system.

Learning objectives

By the end of this module, you should be able to:

  • Design fallback and routing policies across models and non-AI alternatives.
  • Estimate per-request workload/cost and identify context/token drivers.
  • Define reliability objectives that include task success as well as service latency/availability.

Production architecture needs graceful degradation

If the preferred model or retrieval dependency is unavailable, the product should define what remains useful: a smaller model, search-only results, cached approved answer, human escalation or explicit temporary unavailability.

Route by task, not prestige

Some requests need high reasoning quality; others are extraction, classification or lookup. Routing can reduce latency and cost if evaluation proves the smaller path meets the contract.

Caching needs semantic and policy awareness

Cache deterministic embeddings and stable retrieval where appropriate. Response caching is riskier when user identity, authorization, freshness or personalized context changes the answer.

Context growth is an operational risk

Long conversations and broad retrieval increase latency and workload. Summarize state, cap histories, deduplicate evidence and make context budgets observable.

Capacity includes accelerators and external quotas

Local GPU memory/concurrency or remote rate limits can be the bottleneck. Load tests should model prompt/output sizes and tool/retrieval dependencies, not just requests per second.

AI reliability needs outcome objectives

A service can return HTTP 200 quickly while producing unusable answers. Pair operational objectives with evaluated task-success, fallback correctness or groundedness targets for the feature.

Worked examples

Fallback ladder

preferred model + RAG
→ smaller approved model + RAG
→ retrieval/search only
→ human escalation
→ explicit unavailable
  • Every fallback has quality and policy expectations.

Unit economics

request workload ≈ input/context units + output units + embedding/retrieval + tool calls
monthly cost/capacity = request mix × per-path workload
  • Optimize after measuring the real request distribution.
Try it · core lab · account-free

Lab 11 — Build routing and fallback policy

Goal: Design a measurable production decision table rather than a hard-coded 'always use biggest model' rule.

Before you start

  • No paid API or cloud account is required for the core lab.
  • Core labs use Python 3 and local files only.
  • Running case: you are engineering 'Atlas Assist', an internal support assistant over a small approved knowledge base with one read-only inventory tool.
  • Carry requirements, eval cases, threat assumptions and operational evidence forward across modules instead of starting over.
STEP 1

Create a routing table

cat > /tmp/ai-routing.json <<'EOF'
[
 {"task":"classification","route":"small-local","fallback":"rules","p95_ms":700},
 {"task":"grounded_support","route":"quality-model","fallback":"search-only","p95_ms":1800},
 {"task":"privileged_change","route":"none","fallback":"human","p95_ms":0}
]
EOF
STEP 2

Validate explicit fallback

python3 - <<'PY2'
import json
x=json.load(open('/tmp/ai-routing.json'))
assert all(r.get('fallback') for r in x)
assert next(r for r in x if r['task']=='privileged_change')['fallback']=='human'
print('PASS: every route has fallback')
PY2
STEP 3

Add capacity assumptions

For each AI route, record expected context/output size, concurrency, accelerator/API quota and the threshold that triggers load shedding or a fallback route.

Verify

  • Every route has a defined fallback that preserves policy boundaries.
  • At least one high-impact task deliberately avoids autonomous model execution.

Expected outcome

  • A task-routing, fallback and capacity policy tied to measurable SLOs.

Break / fix · diagnose

  • If fallback silently changes authorization or grounding rules, it is unsafe.
  • If capacity testing uses only tiny prompts, add representative context/output distributions.
Production view

Production AI reliability is a property of routing, dependencies, capacity, policy and fallback—not only the model endpoint.

Local real-tool practicum · optional but recommended

Optional practicum — compare local request paths

Run a short and long-context request and record timing/counters; use the result to discuss context budgets, not to declare a universal benchmark.

Prerequisites

  • Ollama installed and reachable on http://127.0.0.1:11434 for the real-tool path.

Resources created

  • Only local disposable files/processes used by the practicum.

Cost / risk

  • No cloud account required; model downloads consume disk and network bandwidth.
TOOL 1
MODEL=${MODEL:-qwen3:8b}; python3 - <<'PY2' > /tmp/long-context.txt
print('approved support evidence ' * 1200)
PY2
TOOL 2
wc -c /tmp/long-context.txt
TOOL 3
echo 'Run one short and one long-context request against your approved local model; record total_duration, prompt_eval_count and eval_count.'

Verify

  • You have comparable request records with model identity and input-size evidence.

Cleanup

  • Remove only the disposable files created for the lab; keep model caches only if you intend to reuse them.

Warnings

  • Do not publish local benchmark numbers as general model rankings; hardware, runtime, quantization and prompt shape materially change results.
Avoid these traps

Common mistakes

  • Using the largest model for every task.
  • Fallbacks that weaken access control or grounding requirements.
  • Caching personalized/authorized answers without the correct cache key and policy.
  • Capacity tests that ignore realistic context and output distributions.
Knowledge check

Can you connect the AI behaviour to a system control?

Concept and scenario checks. Reveal the answer only after you commit to a decision.

1What should a fallback preserve?

Core safety, authorization and evidence contracts, even if capability or presentation degrades.

2Why route by task?

Different tasks can meet their quality target with different model/cost/latency profiles.

3Why can response caching be risky?

Identity, authorization, freshness and personalized context may change what answer is allowed or correct.

4Scenario: the model endpoint is healthy but grounded task success falls below target. Is the AI feature reliable?

No; operational health alone does not satisfy the feature's outcome reliability objective.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.