An AI model is only one part of an application. In this module you learn what the model does, what the surrounding software must control, why answers can vary, and how to define what “good” and “safe” mean before writing code.
Engineering focus: AI systems: behaviour, uncertainty and engineering contracts
An AI model is only one part of an application. In this module you learn what the model does, what the surrounding software must control, why answers can vary, and how to define what “good” and “safe” mean before writing code.
The rest of this module introduces the formal terminology and production controls behind that idea. You are not expected to memorize the vocabulary before using it.
Carry the same support assistant forward: approved KB, source citations, explicit insufficient-evidence fallback and one bounded read-only asset lookup. Reuse prior artifacts so each module adds evidence to one production AI system.
A language model maps context to a probability distribution over possible next tokens. The application around it decides what context is supplied, what tools are exposed, how outputs are validated and what happens when confidence or policy is insufficient.
Treating the model as the entire system hides the controls that make an AI feature dependable: retrieval, schemas, authorization, evaluation, observability and fallback.
A strong AI requirement names the user, the task, the acceptable evidence and the consequence of a wrong answer. 'Add a chatbot' is not an engineering requirement.
For Atlas Assist, the useful contract is narrower: answer internal support questions using approved documents, cite the supporting passages, and refuse or escalate when the evidence is insufficient.
Traditional software can often assert one deterministic output for one input. Generative systems may produce multiple acceptable outputs, so tests need criteria such as factual support, schema validity, task completion, policy compliance and latency rather than exact string equality.
Instructions, conversation history, retrieved evidence and tool results all consume context. More context is not automatically better: irrelevant material raises cost and can distract the model. Context engineering is the deliberate selection, ordering and compression of information needed for the current decision.
Prompt quality matters, but production quality comes from a loop: specify behaviour, build a representative evaluation set, instrument failures, change one part of the system and measure whether the target behaviour improved.
User → application policy → context builder → model → validator → response
↘ tools / retrievalTask: answer approved support questions
Evidence: approved KB only
Output: answer + source IDs + confidence state
Fail closed: if evidence is insufficient, say so and escalate
Success: grounded, useful, policy-compliant responseGoal: Turn an ambiguous assistant request into an explicit, testable engineering contract.
cat > /tmp/atlas-contract.json <<'EOF'
{
"user": "internal support engineer",
"task": "answer approved operational questions",
"allowed_sources": ["kb"],
"must_return": ["answer","source_ids","status"],
"forbidden": ["invented credentials","unsupported claims"],
"fallback": "INSUFFICIENT_EVIDENCE"
}
EOFpython3 - <<'PY2'
import json
x=json.load(open('/tmp/atlas-contract.json'))
required={'user','task','allowed_sources','must_return','forbidden','fallback'}
missing=required-set(x)
assert not missing, missing
assert 'source_ids' in x['must_return']
print('PASS: explicit AI application contract')
PY2Write one example where the assistant should answer and one where it must return INSUFFICIENT_EVIDENCE. These become seed evaluation cases later.
AI teams move faster when model experiments are judged against a stable application contract rather than subjective demos.
Confirm that the model service and application are separate operational surfaces.
curl -fsS http://127.0.0.1:11434/api/versioncurl -fsS http://127.0.0.1:11434/api/tags | python3 -m json.tool | head -80Concept and scenario checks. Reveal the answer only after you commit to a decision.
Because policy, context, retrieval, tools, validation, authorization, evaluation and fallback are application responsibilities around the model.
Multiple outputs can be acceptable, so evaluation must test behavioural criteria such as grounding, schema validity and policy compliance.
The application should follow its explicit fallback or escalation contract rather than inventing an answer.
A grounded application contract and evaluation criterion connecting claims to approved evidence.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.