Mental model
Module 07: Retrieve the right evidence for the right user- Use metadata to constrain retrieval by source, tenant, version, date, language or authorization boundary.
RAG & Knowledge Systems Engineering · Module 07 of 18
Builder
Metadata & Filtering
Use metadata to constrain retrieval by source, tenant, version, date, language or authorization boundary.
Engineering focus: Retrieve the right evidence for the right user
In plain English
What problem are we solving?
Use metadata to constrain retrieval by source, tenant, version, date, language or authorization boundary.
You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.
Running case · Evidence Desk
Improve the same system, module by module.
Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.
Mental model
Understand it before you optimize it.
Metadata is the guardrail around semantic search. It tells the retriever which tenant, version, language, source class or date is eligible.
01User / policy context
02Deterministic filters
03Eligible corpus
04Semantic ranking
05Authorized evidence
Remember
Authorization must be enforced before restricted text can reach the model.
Learning objectives
By the end of this module, you should be able to:
- Design useful metadata fields.
- Apply metadata filters before generation.
- Explain why post-generation filtering is too late for sensitive content.
Metadata adds structure to semantic search
Vectors capture meaning; metadata captures explicit facts such as document type, owner, version, region and permissions.
Authorization belongs before retrieval exposure
A model must never receive chunks the user is not authorized to see.
Filters can hurt recall
Overly strict or incorrect metadata can hide relevant evidence. Test filters as part of retrieval evaluation.
Version metadata prevents stale authority
Track effective dates or supersession relationships so older policies do not outrank current ones simply because their wording is similar.
Engineering depth
Engineering deep dive
- Prefer allow-list filters derived from authenticated user/context.
- Normalize metadata values and test missing/unknown states explicitly.
- Consider pre-filter vs post-filter semantics of the chosen vector/search backend.
- Log authorization decisions without leaking restricted document content.
Worked examples
Filter before semantic exposure
user tenant=RED
ALL INDEX
red-vpn.md
blue-payroll.md
PRE-FILTER tenant=RED
red-vpn.md
SEMANTIC SEARCH
only authorized RED candidates- The unauthorized BLUE text never enters the candidate set.
- This is a deterministic application control.
Diagnostic lens
If this goes wrong, where do you look first?
Symptom
A user receives a semantically relevant document they should not see.
Likely causes
- Authorization applied after retrieval
- Tenant metadata missing
- Filter normalization mismatch
First checks
- Inspect pre-retrieval filters
- Verify tenant/source metadata
- Run a cross-tenant negative test
Engineering principle
Authorization is deterministic application logic, not an LLM judgment.
Try it · core lab · account-free
Lab 07 — Metadata & Filtering
Goal: Filter a corpus by tenant and version before ranking.
Before you start
- Core lab requires only Python 3 and local files unless otherwise stated.
- Running case: build the same evidence-grounded knowledge assistant across the course.
- Keep source IDs and intermediate evidence so each result is inspectable.
- Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.
Run the baseline
python3 - <<'PY2'
docs=[{'id':'a','tenant':'red','active':True},{'id':'b','tenant':'blue','active':True},{'id':'c','tenant':'red','active':False}]
visible=[d for d in docs if d['tenant']=='red' and d['active']]
assert [d['id'] for d in visible]==['a']; print(visible)
PY2Inspect the evidence
Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.
Run a negative authorization test
Pretend the query is from tenant red while the most semantically similar record belongs to tenant blue. Confirm the blue record is removed before ranking/exposure.
Diagnose and restore
Authorization is deterministic application logic, not an LLM judgment. State the failing layer, change only that layer, then repeat the verification checks.
Write the engineering note
Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.
Verify
- Only authorized, active records remain in the candidate set.
- You can explain why filtering must precede prompt construction.
- The baseline and the deliberately broken case produce different, explainable evidence.
- You can name the pipeline layer that owns the failure and the control that repairs it.
Expected outcome
- A reproducible local result plus a short evidence note.
- A short before/after record showing the failure, diagnosis and repaired state.
Break / fix · diagnose
- Inspect the intermediate evidence before changing the model or prompt.
- If a command fails, preserve the error and classify which pipeline layer failed.
- Change one pipeline variable at a time and rerun the same test case.
- Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.
Production view
Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.
Production checklist
What would an engineer verify before shipping this layer?
Authorization pre-filtered
Missing metadata fails safely
Cross-tenant negative tests pass
Avoid these traps
Common beginner and engineering mistakes
- Filtering after the LLM has already seen restricted text.
- Missing version/effective-date metadata.
- Using metadata values without normalization.
Knowledge check
Can you explain the idea and diagnose the system?
1Why are vectors insufficient for authorization?
Semantic similarity does not encode application permissions.
2When must ACL filtering happen?
Before restricted content is exposed to the model or user.
3How can metadata filtering reduce recall?
Incorrect or overly strict filters can remove relevant candidates.
4Why track effective dates?
To distinguish current authoritative sources from stale versions.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.