RAG & Knowledge Systems Engineering · Module 14 of 18

Engineer

Diagnosing Retrieval & Generation Failures

Use traces and evidence inspection to distinguish ingestion, retrieval, reranking, context, generation and citation failures.

Engineering focus: Find the layer that actually failed

In plain English

What problem are we solving?

Use traces and evidence inspection to distinguish ingestion, retrieval, reranking, context, generation and citation failures.

You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.

Running case · Evidence Desk

Improve the same system, module by module.

Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.

Python bridge RAG glossary

Mental model

Understand it before you optimize it.

Troubleshooting RAG is like tracing a packet: follow the evidence through every hop and find the first point where expected state diverges.

01Ingestion trace

02Retrieval trace

03Rerank / filter trace

04Context trace

05Generation / citation trace

Remember

Fix the first broken layer instead of tuning the prompt for every symptom.

Learning objectives

By the end of this module, you should be able to:

Classify common RAG failure modes.
Trace a query through the pipeline.
Change one component at a time and compare eval results.

Failure attribution matters

If parsing lost the relevant paragraph, changing the prompt is wasted effort.

Trace the evidence path

For each query, capture candidates, scores, filters, reranked order, final context and answer citations.

Use controlled experiments

Change chunking, model, top-k or reranking independently where possible so improvements are attributable.

No-answer cases are essential

A system that performs well only when an answer definitely exists is not production-ready.

Engineering depth

Engineering deep dive

Use a failure taxonomy: ingestion → retrieval → filter/rerank → context → generation → citation.
Attach a trace/correlation ID to a query across stages.
Replay production failures against frozen inputs when possible.
Use controlled experiments: one pipeline change per comparison.

Worked examples

Mental model

Module 14: Find the layer that actually failed

Use traces and evidence inspection to distinguish ingestion, retrieval, reranking, context, generation and citation failures.

Failure tree

Question wrong?
  ↓ no
Source ingested? → no: INGESTION
  ↓ yes
Relevant chunk in top-k? → no: RETRIEVAL
  ↓ yes
Survives filter/rerank/context? → no: CONTEXT
  ↓ yes
Answer supported? → no: GENERATION
  ↓ yes
Citation points to support? → no: CITATION

The first failing layer is the best initial repair target.
This prevents random prompt tuning.

Diagnostic lens

If this goes wrong, where do you look first?

Symptom

Teams keep changing prompts but the failure persists.

Likely causes

The fault is upstream
No trace IDs / intermediate evidence
Multiple components change together

First checks

Replay one failing case
Inspect each pipeline hop in order
Change one variable at a time

Engineering principle

Find the earliest divergent layer before tuning downstream behavior.

Try it · core lab · account-free

Lab 14 — Diagnosing Retrieval & Generation Failures

Goal: Classify four synthetic failures into retrieval, context or generation layers.

Before you start

Core lab requires only Python 3 and local files unless otherwise stated.
Running case: build the same evidence-grounded knowledge assistant across the course.
Keep source IDs and intermediate evidence so each result is inspectable.
Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.

STEP 1

Run the baseline

python3 - <<'PY2'
cases={'A':'relevant chunk absent from top-k','B':'relevant chunk retrieved but removed by bad filter','C':'good context supplied but answer invents extra fact','D':'citation points to wrong chunk'}
labels={'A':'retrieval','B':'context/filter','C':'generation','D':'citation'}
for k,v in cases.items(): print(k,labels[k],v)
PY2

STEP 2

Inspect the evidence

Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.

STEP 3

Misdiagnose on purpose

Take a retrieval failure and try to fix it only by changing the generation prompt. Explain why the symptom cannot be repaired downstream.

STEP 4

Diagnose and restore

Find the earliest divergent layer before tuning downstream behavior. State the failing layer, change only that layer, then repeat the verification checks.

STEP 5

Write the engineering note

Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.

Verify

Each failure is assigned to a pipeline layer.
You can state what evidence would confirm the diagnosis.
The baseline and the deliberately broken case produce different, explainable evidence.
You can name the pipeline layer that owns the failure and the control that repairs it.

Expected outcome

A reproducible local result plus a short evidence note.
A short before/after record showing the failure, diagnosis and repaired state.

Break / fix · diagnose

Inspect the intermediate evidence before changing the model or prompt.
If a command fails, preserve the error and classify which pipeline layer failed.
Change one pipeline variable at a time and rerun the same test case.
Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.

Production view

Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.

Production checklist

What would an engineer verify before shipping this layer?

Gate 1

Trace spans pipeline layers

Gate 2

Failure taxonomy used

Gate 3

One-change-at-a-time experiments recorded

Avoid these traps

Common beginner and engineering mistakes

Tweaking prompts for every failure.
Changing multiple pipeline components at once.
Ignoring no-answer and adversarial cases.

Knowledge check

Can you explain the idea and diagnose the system?

1Why is failure classification useful?

It directs engineering effort to the layer that actually failed.

2What should a trace capture?

Candidates, filters, ranks, final context, generated answer and citations.

3Why change one variable at a time?

To attribute improvements or regressions.

4Why include no-answer tests?

To measure abstention and unsupported-answer behaviour.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.