AI threat surfaces
user input · retrieved content · model artifact · tool calls · model output · logs/telemetry · feedback data- Threat model the entire flow and trust boundaries.
AI Engineering Fundamentals · Module 09 of 12
Protect the AI application
Learn how prompt injection, poisoned retrieved content, sensitive-data leakage and tool abuse become application security problems, and how layered controls reduce risk.
Engineering focus: AI security, safety and governance
In plain English
What problem are we solving?
Learn how prompt injection, poisoned retrieved content, sensitive-data leakage and tool abuse become application security problems, and how layered controls reduce risk.
The rest of this module introduces the formal terminology and production controls behind that idea. You are not expected to memorize the vocabulary before using it.
Running case · Atlas Assist
Carry the same support assistant forward: approved KB, source citations, explicit insufficient-evidence fallback and one bounded read-only asset lookup. Reuse prior artifacts so each module adds evidence to one production AI system.
Learning objectives
By the end of this module, you should be able to:
- Threat-model user input, retrieved content, tools, outputs and model/data supply chain.
- Design controls for prompt injection and sensitive information disclosure without pretending prompt text is a complete security boundary.
- Map technical controls to risk ownership, monitoring and human escalation.
Prompt injection is a system security problem
Attackers can place adversarial instructions directly in user input or indirectly in documents/web content. No single prompt reliably solves this; use trust boundaries, least-privilege tools, content isolation, authorization and output/action validation.
Sensitive data can leak through multiple paths
Prompts, retrieval indexes, conversation history, logs, tool results and training/fine-tuning data can all expose secrets or personal information. Minimize data before it reaches the model and control what is logged.
Improper output handling crosses into classic application security
Model output is untrusted input to downstream software. Escape content for HTML/SQL/shell contexts, validate structured fields and never execute generated code or commands merely because they came from a model.
Supply chain includes models and data
Track model source/digest/licence, embedding models, prompt/templates, datasets, packages and retrieval content. A poisoned document or untrusted model artifact can change behaviour without a code change.
Governance connects risk to decisions
NIST AI RMF and its Generative AI Profile provide a risk-management structure. The engineering team still needs system-specific owners, acceptance criteria, monitoring and escalation; framework adoption is not a substitute for controls.
Safety is workload specific
Define harms and unacceptable outcomes for the actual application. A support assistant, code agent and medical workflow have different impact boundaries, review requirements and fallback policies.
Worked examples
Layered injection defense
untrusted content isolation + retrieval ACLs + narrow tools + app authorization + output validation + monitoring + human approval for impact- Do not rely on 'ignore malicious instructions' as the only control.
Try it · core lab · account-free
Lab 09 — Threat-model Atlas Assist
Goal: Create an abuse-case table and verify that controls sit outside the model where appropriate.
Before you start
- No paid API or cloud account is required for the core lab.
- Core labs use Python 3 and local files only.
- Running case: you are engineering 'Atlas Assist', an internal support assistant over a small approved knowledge base with one read-only inventory tool.
- Carry requirements, eval cases, threat assumptions and operational evidence forward across modules instead of starting over.
Create threats
cat > /tmp/ai-threats.json <<'EOF'
[
{"threat":"indirect_prompt_injection","surface":"retrieved_document","control":"treat document as data; narrow tools; app authorization"},
{"threat":"secret_disclosure","surface":"logs","control":"redaction/minimization and restricted retention"},
{"threat":"unsafe_tool_call","surface":"tool_executor","control":"schema + user authorization + confirmation"}
]
EOFCheck every threat has a non-prompt control
python3 - <<'PY2'
import json
x=json.load(open('/tmp/ai-threats.json'))
assert len(x)>=3
assert all('control' in r and len(r['control'])>20 for r in x)
assert any('authorization' in r['control'] for r in x)
print('PASS: layered threat controls')
PY2Add one monitoring signal
For each threat, add an observable signal: denied tool calls, sensitive-data redaction count, suspicious instruction patterns or retrieval-source anomalies. Monitoring is not the prevention control but makes attacks and drift visible.
Verify
- Threats cover at least input/retrieval, tools and data/logging.
- At least one critical action is blocked by application authorization independent of model output.
Expected outcome
- A system-specific threat model with preventive and detective controls.
Break / fix · diagnose
- If every control is a prompt instruction, redesign the trust boundary.
- If telemetry captures complete prompts by default, review privacy and secret leakage.
Production view
AI security combines classic application security with new model/data interaction risks; the controls must surround the model.
Local real-tool practicum · optional but recommended
Optional practicum — adversarial local test set
Run bundled prompt-injection and unsupported-evidence cases through your local pipeline and store the results for Module 08 evaluation.
Prerequisites
- Ollama installed and reachable on http://127.0.0.1:11434 for the real-tool path.
Resources created
- Only local disposable files/processes used by the practicum.
Cost / risk
- No cloud account required; model downloads consume disk and network bandwidth.
unzip -q public/labs/ai/AI-Engineering-Lab-Pack-v0.3.19-r2.zip -d /tmp/ai-sec-packcat /tmp/ai-sec-pack/security/adversarial-cases.jsonlpython3 /tmp/ai-sec-pack/security/check_policy.py /tmp/ai-sec-pack/security/adversarial-cases.jsonlVerify
- The deterministic policy checker identifies the expected trust-boundary decisions.
Cleanup
- Remove only the disposable files created for the lab; keep model caches only if you intend to reuse them.
Warnings
- Do not test prompt injection against production systems or data without authorization and a defined test plan.
Avoid these traps
Common mistakes
- Treating prompt injection as solved by a stronger system prompt.
- Logging full prompts/responses without data classification.
- Executing model output in shell/SQL/HTML without context-specific validation or escaping.
- Downloading models/datasets without provenance and licence review.
Knowledge check
Can you connect the AI behaviour to a system control?
Concept and scenario checks. Reveal the answer only after you commit to a decision.
1Why is prompt injection not only a prompt-writing problem?
Because adversarial instructions can enter through untrusted content and become dangerous when the surrounding system grants tools/data authority.
2How should model output be treated by downstream software?
As untrusted input requiring validation/escaping appropriate to the destination.
3What does AI supply-chain provenance include?
Model/data sources and versions plus prompts, embedding models, libraries and retrieval content lineage.
4Scenario: a retrieved page asks the model to call an admin tool. What is the primary control?
The page remains untrusted data and the application authorization/tool policy must deny unauthorized admin actions.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.