RAG & Knowledge Systems Engineering · Module 13 of 18

Engineer

RAG Evaluation from First Principles

Build a labelled evaluation set and use retrieval metrics plus grounded-answer checks so improvements are measurable rather than anecdotal.

Engineering focus: Measure retrieval and answer quality separately

In plain English

What problem are we solving?

Build a labelled evaluation set and use retrieval metrics plus grounded-answer checks so improvements are measurable rather than anecdotal.

You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.

Running case · Evidence Desk

Improve the same system, module by module.

Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.

Python bridge RAG glossary

Mental model

Understand it before you optimize it.

Evaluation is a test bench with separate gauges. Retrieval, grounding and answer usefulness need different measurements so one score cannot hide the failing layer.

01Labelled cases

02Retriever metrics

03Grounding metrics

04Failure slices

05Regression gate

Remember

Measure components separately before combining them into a release decision.

Learning objectives

By the end of this module, you should be able to:

Create representative query/evidence test cases.
Compute Recall@k and MRR from simple labels.
Separate retrieval metrics from generation metrics.

Evaluate the pipeline in layers

If the correct passage was never retrieved, generation cannot fix that reliably. Measure retrieval before judging answer style.

Recall@k asks whether needed evidence was found

For each query, check whether relevant evidence appears within the top k results.

MRR rewards early relevant results

Mean Reciprocal Rank is higher when the first relevant result appears near the top.

Generation needs separate checks

Faithfulness/groundedness, answer correctness, citation support and abstention behaviour belong to the generation layer.

Engineering depth

Engineering deep dive

Use labelled relevance judgments for retrieval metrics such as Recall@k, Precision@k, MRR or nDCG as appropriate.
Measure grounding/faithfulness separately from retriever performance.
Include negative/no-answer and adversarial cases.
Version datasets, prompts, models and index configuration with every benchmark.

Worked examples

Mental model

Module 13: Measure retrieval and answer quality separately

Build a labelled evaluation set and use retrieval metrics plus grounded-answer checks so improvements are measurable rather than anecdotal.

Measure layers separately

Retriever
  Recall@5      0.91
  MRR           0.84

Generator
  Faithfulness  0.95
  Abstention    0.88

Security
  ACL leakage   0 failures / 120 tests

A high faithfulness score cannot compensate for missing evidence.
Separate gates make regressions diagnosable.

Diagnostic lens

If this goes wrong, where do you look first?

Symptom

A demo looks better, but nobody can prove the system improved.

Likely causes

No labelled dataset
Only end-to-end subjective review
Metrics averaged across different failure types

First checks

Freeze a test set
Measure retrieval and grounding separately
Slice failures by query type

Engineering principle

Evaluation must be reproducible and diagnostic.

Try it · core lab · account-free

Lab 13 — RAG Evaluation from First Principles

Goal: Compute Recall@3 and reciprocal rank on a tiny labelled retrieval set.

Before you start

Core lab requires only Python 3 and local files unless otherwise stated.
Running case: build the same evidence-grounded knowledge assistant across the course.
Keep source IDs and intermediate evidence so each result is inspectable.
Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.

STEP 1

Run the baseline

python3 - <<'PY2'
cases=[({'a'},['x','a','b']),({'c'},['c','d','e']),({'z'},['a','b','c'])]
hits=[]; rr=[]
for rel,ranked in cases:
  positions=[i+1 for i,d in enumerate(ranked[:3]) if d in rel]
  hits.append(bool(positions)); rr.append(1/min(positions) if positions else 0)
print('Recall@3=',sum(hits)/len(hits)); print('MRR=',sum(rr)/len(rr))
PY2

STEP 2

Inspect the evidence

Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.

STEP 3

Expose an aggregate-score blind spot

Create a no-answer case that retrieval cannot satisfy. Show why a good average MRR does not prove correct abstention behaviour.

STEP 4

Diagnose and restore

Evaluation must be reproducible and diagnostic. State the failing layer, change only that layer, then repeat the verification checks.

STEP 5

Write the engineering note

Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.

Verify

Metrics are derived from labelled expected evidence.
A retrieval regression can be detected before involving an LLM.
The baseline and the deliberately broken case produce different, explainable evidence.
You can name the pipeline layer that owns the failure and the control that repairs it.

Expected outcome

A reproducible local result plus a short evidence note.
A short before/after record showing the failure, diagnosis and repaired state.

Break / fix · diagnose

Inspect the intermediate evidence before changing the model or prompt.
If a command fails, preserve the error and classify which pipeline layer failed.
Change one pipeline variable at a time and rerun the same test case.
Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.

Production view

Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.

Production checklist

What would an engineer verify before shipping this layer?

Gate 1

Versioned labelled eval set

Gate 2

Retrieval and generation measured separately

Gate 3

Negative/adversarial slices included

Avoid these traps

Common beginner and engineering mistakes

Evaluating only a handful of demo questions.
Combining retrieval and generation failures into one score.
Optimizing a metric without inspecting failure cases.

Knowledge check

Can you explain the idea and diagnose the system?

1What does Recall@k measure?

Whether relevant evidence appears within the top k retrieved items.

2What does MRR reward?

Placing the first relevant result near the top.

3Why evaluate retrieval separately?

To localize failures before blaming generation.

4What should an eval set represent?

Real query types, difficult cases, no-answer cases and important risk scenarios.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.