Mental model
Module 09: Improve what reaches the model- Use candidate generation, reranking, query rewriting and decomposition to improve evidence quality without hiding the original question.
RAG & Knowledge Systems Engineering · Module 09 of 18
Builder
Reranking & Query Improvement
Use candidate generation, reranking, query rewriting and decomposition to improve evidence quality without hiding the original question.
Engineering focus: Improve what reaches the model
In plain English
What problem are we solving?
Use candidate generation, reranking, query rewriting and decomposition to improve evidence quality without hiding the original question.
You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.
Running case · Evidence Desk
Improve the same system, module by module.
Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.
Mental model
Understand it before you optimize it.
First-stage retrieval casts a wide net; reranking spends more compute on a smaller set. Query rewriting can help search, but the original user intent remains authoritative.
01Original query
02Candidate retrieval
03Optional rewrite / decomposition
04Rerank
05Final evidence
Remember
Never let a rewrite silently become a different user request.
Learning objectives
By the end of this module, you should be able to:
- Explain retrieve-then-rerank architecture.
- Use query rewriting safely while preserving user intent.
- Recognize when a question should be decomposed into subqueries.
Retrieve broadly, rerank precisely
A fast first-stage retriever can collect candidates; a stronger reranker can score them with more context.
Query rewriting is a retrieval aid
Rewrite ambiguous or conversational queries into search-friendly form, but keep the original request for final answer generation and audit.
Decomposition helps multi-part questions
Questions requiring evidence from multiple sources may need separate retrieval steps.
Rerankers do not create authority
They improve relevance ordering; metadata, provenance and policy still determine whether a source is allowed and authoritative.
Engineering depth
Engineering deep dive
- Use first-stage retrieval for recall and second-stage reranking for precision.
- Evaluate query rewriting against intent-preservation tests.
- For decomposition, retain a mapping from sub-query to evidence and final claim.
- Do not let reranker relevance override source authorization/authority.
Worked examples
Retrieve wide, rerank narrow
original query
↓
retrieve top 20
↓
rerank with query + full candidate text
↓
keep top 5
↓
authority/context policy
↓
send 2–3 chunks to model- Candidate recall and final precision are separate concerns.
- Original intent stays available throughout.
Diagnostic lens
If this goes wrong, where do you look first?
Symptom
Retrieved candidates are relevant, but the final evidence set is poor.
Likely causes
- Reranker feature/model mismatch
- Rewrite changed intent
- Decomposition omitted a sub-question
First checks
- Compare pre/post rerank order
- Log original and rewritten query
- Check evidence per sub-question
Engineering principle
Preserve intent and trace every transformation.
Try it · core lab · account-free
Lab 09 — Reranking & Query Improvement
Goal: Rerank a candidate list using simple rule-based evidence features and preserve original query text.
Before you start
- Core lab requires only Python 3 and local files unless otherwise stated.
- Running case: build the same evidence-grounded knowledge assistant across the course.
- Keep source IDs and intermediate evidence so each result is inspectable.
- Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.
Run the baseline
python3 - <<'PY2'
q='How do I reset VPN access?'
c=[('old',1,False),('vpn-current',2,True),('generic',1,True)]
ranked=sorted(c,key=lambda x:(x[2],x[1]),reverse=True)
print({'original_query':q,'ranked':ranked})
PY2Inspect the evidence
Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.
Catch intent drift
Write a bad rewrite that changes 'reset VPN access' into 'disable VPN access'. Compare it to the original and mark the rewrite invalid.
Diagnose and restore
Preserve intent and trace every transformation. State the failing layer, change only that layer, then repeat the verification checks.
Write the engineering note
Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.
Verify
- The original query remains available.
- Current/authoritative evidence can be promoted by explicit features.
- The baseline and the deliberately broken case produce different, explainable evidence.
- You can name the pipeline layer that owns the failure and the control that repairs it.
Expected outcome
- A reproducible local result plus a short evidence note.
- A short before/after record showing the failure, diagnosis and repaired state.
Break / fix · diagnose
- Inspect the intermediate evidence before changing the model or prompt.
- If a command fails, preserve the error and classify which pipeline layer failed.
- Change one pipeline variable at a time and rerun the same test case.
- Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.
Production view
Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.
Production checklist
What would an engineer verify before shipping this layer?
Original query preserved
Rerank before/after trace retained
Rewrite/decomposition intent tests pass
Avoid these traps
Common beginner and engineering mistakes
- Letting a rewrite silently change user intent.
- Using reranker score as authorization.
- Reranking without logging before/after candidates.
Knowledge check
Can you explain the idea and diagnose the system?
1What is reranking?
Re-scoring an initial candidate set with a stronger relevance model or features.
2Why preserve the original query?
For intent fidelity, audit and final answer construction.
3When is decomposition useful?
When a question needs evidence from multiple subproblems or sources.
4Does reranking replace metadata authority?
No. Relevance and authority are different dimensions.
My Learning
Record your progress for this module.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.