Mental model
Module 15: Keep the index synchronized with reality- Design ingestion, updates, re-indexing, deletion and freshness checks so the knowledge system does not quietly serve stale evidence.
RAG & Knowledge Systems Engineering · Module 15 of 18
Production
Freshness & Knowledge Lifecycle
Design ingestion, updates, re-indexing, deletion and freshness checks so the knowledge system does not quietly serve stale evidence.
Engineering focus: Keep the index synchronized with reality
In plain English
What problem are we solving?
Design ingestion, updates, re-indexing, deletion and freshness checks so the knowledge system does not quietly serve stale evidence.
You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.
Running case · Evidence Desk
Improve the same system, module by module.
Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.
Mental model
Understand it before you optimize it.
A knowledge index is a deployed data product, not a one-time upload. Sources are created, updated, superseded, deleted and re-indexed over time.
01Discover change
02Upsert / delete
03Re-index
04Verify freshness
05Retire old evidence
Remember
Create without delete is not a complete knowledge lifecycle.
Learning objectives
By the end of this module, you should be able to:
- Design idempotent upsert and deletion flows.
- Track source-to-index freshness.
- Explain how model/preprocessing changes trigger re-indexing.
Knowledge changes after launch
Policies are replaced, runbooks are corrected and access rights change. The index must reflect those changes predictably.
Deletion is a first-class operation
Removing a source from the repository is not enough if its vectors remain searchable.
Freshness needs observability
Track source modified time, ingestion time, indexed version and pipeline failures.
Re-index with provenance
When embeddings or chunking change, record the new index generation and support controlled cutover/rollback.
Engineering depth
Engineering deep dive
- Model create/update/supersede/delete and legal retention separately.
- Use idempotent upserts and tombstones/deletes with verification.
- Track index generation/version and ingestion lag.
- Define freshness SLOs appropriate to the knowledge domain.
Worked examples
Knowledge lifecycle state machine
DISCOVERED -> PARSED -> INDEXED -> ACTIVE
|
+-> SUPERSEDED -> RETIRED
+-> DELETED -> PURGED/RETAINED-PER-POLICY
Every transition emits version + timestamp + outcome.- Freshness is observable state, not an assumption.
- Deletion semantics must be designed before production.
Diagnostic lens
If this goes wrong, where do you look first?
Symptom
The assistant keeps returning an old policy after an update.
Likely causes
- Old chunks remain indexed
- Update job failed partially
- Cache/index version is stale
First checks
- Compare source version to index manifest
- Search directly for superseded chunk IDs
- Test deletion propagation
Engineering principle
Freshness requires observable update and delete paths.
Try it · core lab · account-free
Lab 15 — Freshness & Knowledge Lifecycle
Goal: Simulate idempotent upsert and deletion in a tiny JSON index manifest.
Before you start
- Core lab requires only Python 3 and local files unless otherwise stated.
- Running case: build the same evidence-grounded knowledge assistant across the course.
- Keep source IDs and intermediate evidence so each result is inspectable.
- Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.
Run the baseline
python3 - <<'PY2'
index={'doc1':{'version':1},'doc2':{'version':1}}
index['doc1']={'version':2}; index.pop('doc2',None)
assert index=={'doc1':{'version':2}}; print(index)
PY2Inspect the evidence
Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.
Leave a stale record
Update a source but intentionally keep the previous indexed version active. Detect the duplicate authority and then retire/delete the stale version.
Diagnose and restore
Freshness requires observable update and delete paths. State the failing layer, change only that layer, then repeat the verification checks.
Write the engineering note
Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.
Verify
- Updates replace previous indexed versions deterministically.
- Deleted sources disappear from the index manifest.
- The baseline and the deliberately broken case produce different, explainable evidence.
- You can name the pipeline layer that owns the failure and the control that repairs it.
Expected outcome
- A reproducible local result plus a short evidence note.
- A short before/after record showing the failure, diagnosis and repaired state.
Break / fix · diagnose
- Inspect the intermediate evidence before changing the model or prompt.
- If a command fails, preserve the error and classify which pipeline layer failed.
- Change one pipeline variable at a time and rerun the same test case.
- Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.
Production view
Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.
Production checklist
What would an engineer verify before shipping this layer?
Upsert and delete paths tested
Freshness/index version observable
Stale-source regression case passes
Avoid these traps
Common beginner and engineering mistakes
- Only supporting create, not delete.
- Re-indexing without version identifiers.
- Serving stale content after ingestion failures.
Knowledge check
Can you explain the idea and diagnose the system?
1Why must deletion reach the vector index?
Otherwise removed content may remain retrievable.
2What should freshness telemetry compare?
Source version/time against indexed version/time.
3When might full re-indexing be required?
When embeddings, chunking or major preprocessing changes.
4Why version an index generation?
To support reproducibility, controlled cutover and rollback.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.