Mental model
Module 01: Why RAG exists- Understand why an LLM can sound confident yet still need external evidence, and learn the simplest mental model for retrieval-augmented generation.
RAG & Knowledge Systems Engineering · Module 01 of 18
Foundation
Why LLMs Need External Knowledge
Understand why an LLM can sound confident yet still need external evidence, and learn the simplest mental model for retrieval-augmented generation.
Engineering focus: Why RAG exists
In plain English
What problem are we solving?
Understand why an LLM can sound confident yet still need external evidence, and learn the simplest mental model for retrieval-augmented generation.
You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.
Running case · Evidence Desk
Improve the same system, module by module.
Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.
Mental model
Understand it before you optimize it.
Think of the LLM as a capable colleague with no guaranteed access to your latest handbook. RAG gives that colleague a controlled evidence pack before they answer.
01Question
02Retrieve evidence
03Assemble context
04Generate answer
05Cite or abstain
Remember
RAG changes what evidence the model can see at answer time; it does not retrain the model.
Learning objectives
By the end of this module, you should be able to:
- Explain the difference between model knowledge and retrieved knowledge.
- Describe the basic RAG flow in plain language.
- Recognize when RAG is appropriate—and when it is not.
The core problem
A language model generates from patterns learned during training. It may not know your private documents, may not know the newest facts, and can still produce fluent unsupported claims.
The simplest RAG mental model
RAG means: find relevant evidence first, then ask the model to answer using that evidence.
RAG is not magic
Retrieval can fail, the wrong passage can be selected, and the model can still ignore good evidence. Production RAG is a pipeline that must be measured end to end.
When not to use RAG
Use deterministic lookups, databases or tools when the task requires exact structured facts, transactions or authorization rather than fuzzy document reasoning.
Engineering depth
Engineering deep dive
- Differentiate parametric knowledge, retrieved knowledge and authoritative structured data.
- Define an answerability boundary: what the corpus can and cannot support.
- Prefer deterministic tools/APIs for exact transactional facts and RAG for evidence-grounded document reasoning.
- Track unsupported-answer rate, not only answered-query quality.
Worked examples
Evidence-first trace
Question: "What port does the API use?"
1. Search approved knowledge sources
2. Retrieve: network.md#api → "The API listens on TCP 8080."
3. Build context with source ID
4. Answer: "TCP 8080"
5. Cite: network.md#api
Unanswerable query → INSUFFICIENT_EVIDENCE- The model is not asked to invent the port.
- The same contract handles known and unknown questions.
Diagnostic lens
If this goes wrong, where do you look first?
Symptom
The answer sounds plausible but no source supports it.
Likely causes
- The question bypassed retrieval
- No relevant source exists
- The generator answered from model memory
First checks
- Inspect retrieved source IDs
- Repeat with a known answerable question
- Test an intentionally unanswerable question
Engineering principle
Establish evidence before discussing prompt quality.
Try it · core lab · account-free
Lab 01 — Why LLMs Need External Knowledge
Goal: Create a three-document mini knowledge base and manually retrieve the correct source before answering.
Before you start
- Core lab requires only Python 3 and local files unless otherwise stated.
- Running case: build the same evidence-grounded knowledge assistant across the course.
- Keep source IDs and intermediate evidence so each result is inspectable.
- Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.
Run the baseline
printf '%s\n' 'VPN resets require MFA.' 'Payroll closes Friday.' 'API listens on 8080.' > /tmp/rag-kb.txt
grep -n '8080' /tmp/rag-kb.txtInspect the evidence
Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.
Break it deliberately
Ask: 'What is the payroll account number?' The mini corpus has no answer. Record what a forced-answer system would be tempted to do.
Diagnose and restore
Establish evidence before discussing prompt quality. State the failing layer, change only that layer, then repeat the verification checks.
Write the engineering note
Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.
Verify
- The matching line is retrieved before any model is involved.
- You can explain why evidence lookup and generation are separate stages.
- The baseline and the deliberately broken case produce different, explainable evidence.
- You can name the pipeline layer that owns the failure and the control that repairs it.
Expected outcome
- A reproducible local result plus a short evidence note.
- A short before/after record showing the failure, diagnosis and repaired state.
Break / fix · diagnose
- Inspect the intermediate evidence before changing the model or prompt.
- If a command fails, preserve the error and classify which pipeline layer failed.
- Change one pipeline variable at a time and rerun the same test case.
- Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.
Production view
Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.
Production checklist
What would an engineer verify before shipping this layer?
Corpus answerability boundary documented
Unsupported questions tested
Structured-data/API alternatives identified where appropriate
Avoid these traps
Common beginner and engineering mistakes
- Treating RAG as model training.
- Assuming retrieval guarantees truth.
- Using RAG for data that should come from an authoritative API.
Knowledge check
Can you explain the idea and diagnose the system?
1Why can an LLM need RAG even if it is capable?
Because its internal parameters may not contain current, private or authoritative evidence needed for the task.
2What is the simplest RAG sequence?
Retrieve relevant evidence, place it in context, then generate an answer constrained by that evidence.
3Does RAG eliminate hallucinations?
No. Retrieval and generation can each fail, so the pipeline must be evaluated.
4When is an API lookup often better than RAG?
When the answer is structured, exact, transactional or authorization-sensitive.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.