Gnu Group Learning AcademyGnu Group ↗
AI Engineering Fundamentals · Module 12 of 12

Build and validate a production AI system

Bring the complete engineering chain together for Atlas Assist: requirements, model choice, RAG, tools, evaluation, security, observability, reliability and a release decision supported by evidence.

Engineering focus: Production AI engineering capstone

In plain English

What problem are we solving?

Bring the complete engineering chain together for Atlas Assist: requirements, model choice, RAG, tools, evaluation, security, observability, reliability and a release decision supported by evidence.

The rest of this module introduces the formal terminology and production controls behind that idea. You are not expected to memorize the vocabulary before using it.

Running case · Atlas Assist

Carry the same support assistant forward: approved KB, source citations, explicit insufficient-evidence fallback and one bounded read-only asset lookup. Reuse prior artifacts so each module adds evidence to one production AI system.

Learning objectives

By the end of this module, you should be able to:

  • Assemble one coherent AI architecture with explicit trust boundaries and fallbacks.
  • Produce evaluation, security and operational evidence sufficient to defend a release decision.
  • Explain residual risk and what evidence would trigger rollback or human escalation.

The capstone is about connected evidence

A polished chat UI is not sufficient. The deliverable must show how the user contract flows into retrieval, model/tool policy, evaluation gates, security controls and runtime observability.

Promotion is a decision, not a demo

State the baseline and candidate, blocking eval slices, operational budgets, unresolved risks and rollback/fallback plan. If evidence is insufficient, 'do not promote' is a valid engineering outcome.

Trust boundaries must be visible

Label user input, retrieved data, model output and tool proposals as untrusted until validated. Show where document authorization, user authorization and human approval occur.

Evaluation must include adversarial and abstention cases

A capstone that tests only answerable happy paths cannot demonstrate production readiness.

Operations must be diagnosable

Show what trace/metric evidence would tell an on-call engineer whether a failure is retrieval, model inference, tool execution, validation, capacity or policy.

Residual risk belongs in the design

Document what the system does not guarantee and where humans remain responsible. World-class engineering is explicit about limits.

Worked examples

Capstone evidence chain

application contract
→ model/routing decision
→ prompt/context contract
→ retrieval + citations
→ structured output / tools
→ eval baseline + candidate
→ threat model + guardrails
→ telemetry + SLO/fallback
→ PROMOTE | HOLD | ROLLBACK
  • Every arrow must be supported by an artifact or test.

Minimum release record

model/runtime identity
prompt/config version
retrieval/index version
eval dataset version + results
security critical-slice results
latency/capacity evidence
known limitations
rollback/fallback owner
  • This is the evidence needed to explain a future regression.
Try it · core lab · account-free

Lab 12 — Build and defend the Atlas Assist release candidate

Goal: Produce a small production-shaped evidence pack and run the bundled validator.

Before you start

  • No paid API or cloud account is required for the core lab.
  • Core labs use Python 3 and local files only.
  • Running case: you are engineering 'Atlas Assist', an internal support assistant over a small approved knowledge base with one read-only inventory tool.
  • Carry requirements, eval cases, threat assumptions and operational evidence forward across modules instead of starting over.
STEP 1

Download/extract the lab pack

rm -rf /tmp/atlas-ai-capstone && mkdir -p /tmp/atlas-ai-capstone
unzip -q public/labs/ai/AI-Engineering-Lab-Pack-v0.3.19-r2.zip -d /tmp/atlas-ai-capstone
find /tmp/atlas-ai-capstone -maxdepth 2 -type f | sort
STEP 2

Create your release decision

Complete capstone/release-decision.json with model route, fallback, eval thresholds, security blockers, operational budget and PROMOTE/HOLD/ROLLBACK decision.

STEP 3

Run validator

python3 /tmp/atlas-ai-capstone/capstone/validate_release.py /tmp/atlas-ai-capstone/capstone/release-decision.json
STEP 4

Defend the decision

Write 200–300 words explaining the highest residual risk, why your eval set is representative enough for this stage, and which production signal would trigger rollback or human escalation.

Verify

  • The release decision references evaluation, security and operational evidence.
  • Fallback and rollback preserve authorization/evidence constraints.
  • The design distinguishes model output from trusted application decisions.

Expected outcome

  • A defensible AI release evidence pack and explicit promotion decision.

Break / fix · diagnose

  • If the validator passes but your evidence is weak, HOLD is still the correct decision.
  • If the design cannot say what happens when retrieval/model/tool is unavailable, add a fallback before promotion.
Production view

Production AI systems need the same disciplined release evidence as other critical software, plus explicit evaluation of variable model behaviour.

Local real-tool practicum · optional but recommended

Optional practicum — end-to-end local evidence run

Use an approved local chat model and embedding model to generate real outputs for the capstone eval set, while keeping the deterministic validator as the policy gate.

Prerequisites

  • Ollama installed and reachable on http://127.0.0.1:11434 for the real-tool path.

Resources created

  • Only local disposable files/processes used by the practicum.

Cost / risk

  • No cloud account required; model downloads consume disk and network bandwidth.
TOOL 1
MODEL=${MODEL:-qwen3:8b}; EMBED_MODEL=${EMBED_MODEL:-embeddinggemma}; echo "chat=$MODEL embed=$EMBED_MODEL"
TOOL 2
curl -fsS http://127.0.0.1:11434/api/version
TOOL 3
python3 /tmp/atlas-ai-capstone/evals/evaluate.py /tmp/atlas-ai-capstone/evals/sample-results.jsonl

Verify

  • You can attach model/runtime identity and eval output to the release evidence without changing the decision contract.

Cleanup

  • Remove only the disposable files created for the lab; keep model caches only if you intend to reuse them.

Warnings

  • A local model run is optional and hardware-dependent; the certificate assessment must not require a specific vendor/model or GPU.
Capstone rubric

Score connected engineering evidence—not UI polish.

Application contract & architecture · 20%

User task, evidence boundary, output/fallback contract and labelled trust boundaries.

Retrieval & grounding · 15%

Measured retrieval cases, source attribution, missing/conflicting evidence behaviour.

Tools & workflow safety · 15%

Bounded tools, authorization, idempotency/stop rules and approval points.

Evaluation & regression · 20%

Representative dataset, baseline/candidate results, critical slices and decision thresholds.

Security & governance · 15%

Threat model, data minimization, injection/tool/output controls and residual risk.

Operations & release decision · 15%

Telemetry/SLO/cost/fallback evidence plus PROMOTE/HOLD/ROLLBACK rationale.

Avoid these traps

Common mistakes

  • Treating the capstone validator as proof of production safety.
  • Using only happy-path evaluation cases.
  • Leaving tool authorization or document access implicit.
  • Shipping without explicit rollback/fallback and residual-risk ownership.
Knowledge check

Can you connect the AI behaviour to a system control?

Concept and scenario checks. Reveal the answer only after you commit to a decision.

1What makes the capstone production-shaped?

It connects contract, evidence, evaluation, security, operations and a release decision rather than demonstrating isolated model output.

2Is HOLD a valid capstone outcome?

Yes, when evidence does not meet the defined promotion contract.

3What must remain outside model authority?

Security/authorization, release policy, hard budgets and high-impact approval decisions.

4Scenario: all functional evals pass but the critical injection slice fails. What is the release decision?

HOLD or rollback according to policy; critical security slices block promotion despite functional success.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Attempt the core lab and knowledge checks before marking the module complete.