Mental model
Module 12: Know which source should win- Handle stale documents, superseded policies and conflicting evidence using explicit source authority and lifecycle metadata.
RAG & Knowledge Systems Engineering · Module 12 of 18
Engineer
Provenance, Authority & Conflicting Sources
Handle stale documents, superseded policies and conflicting evidence using explicit source authority and lifecycle metadata.
Engineering focus: Know which source should win
In plain English
What problem are we solving?
Handle stale documents, superseded policies and conflicting evidence using explicit source authority and lifecycle metadata.
You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.
Running case · Evidence Desk
Improve the same system, module by module.
Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.
Mental model
Understand it before you optimize it.
Relevance answers “is this about the question?” Authority answers “should this source win?” Production systems need both.
01Relevant sources
02Version / status / owner
03Conflict detection
04Authority policy
05Resolved or unresolved state
Remember
A high similarity score cannot override a superseded or unauthorized source.
Learning objectives
By the end of this module, you should be able to:
- Model source authority separately from semantic relevance.
- Detect stale or superseded evidence.
- Return conflict states instead of silently choosing.
Relevant does not mean authoritative
An old policy can be semantically perfect and still be wrong for today's decision.
Authority needs metadata
Owner, version, effective date, status and supersedes relationships help determine which source governs.
Conflicts are data
If two authoritative sources disagree, the system should surface the conflict or route for review rather than hide uncertainty.
Provenance supports reproducibility
Record which exact source versions and chunks produced an answer.
Engineering depth
Engineering deep dive
- Define authority rules using source class, owner, status, effective time and jurisdiction where relevant.
- Retain superseded documents for audit while excluding them from authoritative answer paths.
- Surface unresolved conflict instead of fabricating reconciliation.
- Record the authority decision alongside retrieval evidence.
Worked examples
Conflict is an explicit state
old-policy.md status=superseded effective=2025-01-01
new-policy.md status=active effective=2026-08-01
Resolution: new-policy.md authoritative
If two ACTIVE authoritative sources conflict:
status = AUTHORITY_CONFLICT- Status/effective date can resolve some conflicts.
- Unresolved authoritative conflict should be surfaced, not blended.
Diagnostic lens
If this goes wrong, where do you look first?
Symptom
Two policies answer the same question differently.
Likely causes
- Old source not marked superseded
- Authority rules missing
- Conflict silently merged
First checks
- Inspect status/effective dates
- Compare owners/source classes
- Force an unresolved-conflict test
Engineering principle
Conflict is a first-class state, not something to hide with similarity.
Try it · core lab · account-free
Lab 12 — Provenance, Authority & Conflicting Sources
Goal: Rank current policy over superseded policy and detect unresolved conflicts.
Before you start
- Core lab requires only Python 3 and local files unless otherwise stated.
- Running case: build the same evidence-grounded knowledge assistant across the course.
- Keep source IDs and intermediate evidence so each result is inspectable.
- Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.
Run the baseline
python3 - <<'PY2'
docs=[{'id':'p1','effective':'2025-01-01','status':'superseded'},{'id':'p2','effective':'2026-08-01','status':'active'}]
active=[d for d in docs if d['status']=='active']; assert len(active)==1; print(active[0])
PY2Inspect the evidence
Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.
Create an unresolved conflict
Mark two contradictory policies as active and equally authoritative. The correct state is AUTHORITY_CONFLICT, not a guessed winner.
Diagnose and restore
Conflict is a first-class state, not something to hide with similarity. State the failing layer, change only that layer, then repeat the verification checks.
Write the engineering note
Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.
Verify
- Superseded evidence is not treated as current authority.
- The system can represent conflict explicitly.
- The baseline and the deliberately broken case produce different, explainable evidence.
- You can name the pipeline layer that owns the failure and the control that repairs it.
Expected outcome
- A reproducible local result plus a short evidence note.
- A short before/after record showing the failure, diagnosis and repaired state.
Break / fix · diagnose
- Inspect the intermediate evidence before changing the model or prompt.
- If a command fails, preserve the error and classify which pipeline layer failed.
- Change one pipeline variable at a time and rerun the same test case.
- Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.
Production view
Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.
Production checklist
What would an engineer verify before shipping this layer?
Authority policy explicit
Superseded sources excluded from authoritative path
Conflict state supported
Avoid these traps
Common beginner and engineering mistakes
- Letting similarity score decide policy authority.
- Deleting old versions without keeping audit history.
- Silently merging contradictory facts.
Knowledge check
Can you explain the idea and diagnose the system?
1Why separate authority from relevance?
A highly relevant source may be stale or non-authoritative.
2What metadata helps resolve policy versions?
Status, effective date, owner, version and supersession links.
3What should happen with unresolved authoritative conflict?
Surface or escalate the conflict rather than fabricate certainty.
4Why keep exact source version in evidence logs?
For reproducibility and audit.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.