Mental model
Module 06: Retrieve semantically similar chunks- Build a small vector index, understand top-k retrieval, and learn why nearest-neighbour search needs filters and evaluation.
RAG & Knowledge Systems Engineering · Module 06 of 18
Builder
Vector Search
Build a small vector index, understand top-k retrieval, and learn why nearest-neighbour search needs filters and evaluation.
Engineering focus: Retrieve semantically similar chunks
In plain English
What problem are we solving?
Build a small vector index, understand top-k retrieval, and learn why nearest-neighbour search needs filters and evaluation.
You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.
Running case · Evidence Desk
Improve the same system, module by module.
Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.
Mental model
Understand it before you optimize it.
Vector search is a candidate finder. top-k controls how many neighbours you inspect before later filtering, reranking and context construction.
01Query vector
02Nearest-neighbour search
03Top-k candidates
04Filters / rerank
05Inspect text
Remember
A vector index retrieves candidates; it does not decide authority or permission.
Learning objectives
By the end of this module, you should be able to:
- Explain top-k nearest-neighbour retrieval.
- Distinguish exact similarity from approximate nearest-neighbour indexing.
- Inspect retrieved chunks rather than trusting scores alone.
Top-k is a design parameter
The retriever returns the k most similar items. Too small can miss evidence; too large can flood context with noise.
Indexes trade speed and recall
Approximate nearest-neighbour indexes improve scale by trading exactness for speed and memory efficiency.
Inspect the text
Always retain the ability to print retrieved chunks, scores and metadata. Retrieval must be observable.
Filter before or after search carefully
Metadata filtering changes the candidate set and can affect both correctness and performance.
Engineering depth
Engineering deep dive
- Understand exact vs approximate nearest-neighbour trade-offs: latency, memory and recall.
- Tune top-k as candidate-generation recall, then let later stages reduce noise.
- Inspect score distributions rather than assuming one global threshold transfers across corpora.
- Benchmark latency and recall at realistic corpus sizes.
Worked examples
Top-k as candidate recall
k=1 -> [network-summary] # misses exact procedure
k=3 -> [network-summary, api-runbook, dns-note]
rerank -> [api-runbook, network-summary]
context -> [api-runbook]- Increase k to improve candidate recall, then control noise later.
- Do not equate top-k with final context size.
Diagnostic lens
If this goes wrong, where do you look first?
Symptom
Relevant evidence exists but is absent from top-k.
Likely causes
- k too small
- Approximate index recall issue
- Query vector does not represent intent
First checks
- Increase k temporarily
- Compare exact vs approximate search if available
- Inspect candidate scores and text
Engineering principle
Tune candidate generation with labelled retrieval cases.
Try it · core lab · account-free
Lab 06 — Vector Search
Goal: Implement a tiny in-memory top-k vector search and inspect ranked results.
Before you start
- Core lab requires only Python 3 and local files unless otherwise stated.
- Running case: build the same evidence-grounded knowledge assistant across the course.
- Keep source IDs and intermediate evidence so each result is inspectable.
- Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.
Run the baseline
python3 - <<'PY2'
from math import sqrt
def cos(a,b): return sum(x*y for x,y in zip(a,b))/(sqrt(sum(x*x for x in a))*sqrt(sum(y*y for y in b)))
q=[1,0]; docs={'network':[.95,.05],'payroll':[.1,.9],'api':[.8,.2]}
print(sorted(((cos(q,v),k) for k,v in docs.items()),reverse=True)[:2])
PY2Inspect the evidence
Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.
Make top-k fail
Set k=1 so a relevant second-ranked candidate is excluded, then repeat with k=2. Record the recall/noise trade-off.
Diagnose and restore
Tune candidate generation with labelled retrieval cases. State the failing layer, change only that layer, then repeat the verification checks.
Write the engineering note
Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.
Verify
- The semantically closest items rank first.
- You can explain the effect of changing k.
- The baseline and the deliberately broken case produce different, explainable evidence.
- You can name the pipeline layer that owns the failure and the control that repairs it.
Expected outcome
- A reproducible local result plus a short evidence note.
- A short before/after record showing the failure, diagnosis and repaired state.
Break / fix · diagnose
- Inspect the intermediate evidence before changing the model or prompt.
- If a command fails, preserve the error and classify which pipeline layer failed.
- Change one pipeline variable at a time and rerun the same test case.
- Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.
Production view
Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.
Production checklist
What would an engineer verify before shipping this layer?
Top-k justified by retrieval eval
Latency and recall measured
Raw text inspected in regression cases
Avoid these traps
Common beginner and engineering mistakes
- Sending every top-k result to the LLM without inspection.
- Assuming a vector database solves access control.
- Ignoring retrieval latency as corpus size grows.
Knowledge check
Can you explain the idea and diagnose the system?
1What does top-k control?
How many nearest candidates are returned.
2Why can a larger k reduce answer quality?
It can introduce irrelevant or conflicting context.
3What is ANN search?
Approximate nearest-neighbour search that trades some exactness for scalable speed.
4What should retrieval logs include?
Query identity, returned source/chunk IDs, scores, filters and latency where appropriate.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.