RAG & Knowledge Systems Engineering · Module 18 of 18

Production

Production Knowledge-System Capstone

Integrate ingestion, retrieval, reranking, grounding, evaluation, security, freshness and observability into one production-shaped knowledge assistant.

Engineering focus: Build, evaluate, secure and operate the complete system

In plain English

What problem are we solving?

Integrate ingestion, retrieval, reranking, grounding, evaluation, security, freshness and observability into one production-shaped knowledge assistant.

You are not expected to know the engineering vocabulary before using the idea. Formal terminology comes after the mental model.

Running case · Evidence Desk

Improve the same system, module by module.

Carry one evidence-grounded knowledge assistant through all 18 modules. Preserve source IDs, retrieval evidence, evaluation cases, authority decisions and security assumptions so every improvement is measurable.

Python bridge RAG glossary

Mental model

Understand it before you optimize it.

The capstone is a release review, not a demo. A production-shaped system earns promotion only when evidence shows retrieval, grounding, security and lifecycle gates pass.

01Build

02Evaluate

03Attack

04Observe

05Review evidence

06PROMOTE or HOLD

Remember

A polished answer is not a release criterion; reproducible evidence is.

Learning objectives

By the end of this module, you should be able to:

Assemble an end-to-end evidence-grounded assistant.
Demonstrate retrieval and grounded-answer evaluation.
Produce an operational handover with known risks and rollback plan.

Production quality is system quality

A good model cannot compensate for stale data, broken authorization, weak evaluation or unobservable retrieval.

Evidence before promotion

The capstone should generate a repeatable evaluation report covering retrieval, citations, abstention, security and latency.

Operate the knowledge lifecycle

Document how new sources are added, old sources removed, indexes versioned and incidents diagnosed.

Know when to hold release

If critical security, authority or grounding tests fail, the correct engineering decision is to stop promotion.

Engineering depth

Engineering deep dive

Define release gates for retrieval, grounding, security, freshness, latency/cost and operability.
Retain a versioned evaluation dataset and evidence dossier.
Exercise rollback/hold paths—not just the happy-path promote decision.
Operational handover should include runbooks, ownership, SLOs and known failure modes.

Worked examples

Mental model

Module 18: Build, evaluate, secure and operate the complete system

Integrate ingestion, retrieval, reranking, grounding, evaluation, security, freshness and observability into one production-shaped knowledge assistant.

Release gate matrix

Retrieval regression       PASS
Grounding / citations       PASS
No-answer behaviour         PASS
ACL isolation               PASS
Injection / poisoning       PASS
Freshness / delete          PASS
Latency / cost              PASS
Rollback evidence           PASS

DECISION: PROMOTE

Each gate links to evidence.
Any critical failure changes the decision to HOLD.

Diagnostic lens

If this goes wrong, where do you look first?

Symptom

The system demos well but release risk is unclear.

Likely causes

Acceptance gates are vague
No adversarial tests
No rollback/evidence dossier

First checks

Review retrieval/grounding/security gates
Run hold-path test
Verify versioned evidence bundle

Engineering principle

Promotion is an engineering decision backed by explicit gates.

Try it · core lab · account-free

Lab 18 — Production Knowledge-System Capstone

Goal: Build a release dossier for the running course project: architecture, eval metrics, adversarial tests, evidence trace and promote/hold decision.

Before you start

Core lab requires only Python 3 and local files unless otherwise stated.
Running case: build the same evidence-grounded knowledge assistant across the course.
Keep source IDs and intermediate evidence so each result is inspectable.
Reflection: Write one sentence describing what this step proves and one sentence describing what it does not prove.

STEP 1

Run the baseline

python3 - <<'PY2'
report={'retrieval_eval':'PASS','grounding_eval':'PASS','acl_eval':'PASS','injection_eval':'PASS','freshness_eval':'PASS','decision':'PROMOTE'}
critical=['retrieval_eval','grounding_eval','acl_eval','injection_eval']
assert all(report[x]=='PASS' for x in critical)
print(report)
PY2

STEP 2

Inspect the evidence

Record the intermediate source IDs, ranks, metadata or state produced by the baseline. Explain which pipeline layer this evidence belongs to.

STEP 3

Exercise the HOLD path

Flip one critical gate—such as ACL isolation—to FAIL. Confirm the release decision becomes HOLD even if every quality metric is strong.

STEP 4

Diagnose and restore

Promotion is an engineering decision backed by explicit gates. State the failing layer, change only that layer, then repeat the verification checks.

STEP 5

Write the engineering note

Record: hypothesis → evidence → change → observed result → remaining limitation. Keep this note with the running Evidence Desk project.

Verify

The release decision is tied to explicit evidence.
Critical security/grounding failures block promotion.
The baseline and the deliberately broken case produce different, explainable evidence.
You can name the pipeline layer that owns the failure and the control that repairs it.

Expected outcome

A reproducible local result plus a short evidence note.
A short before/after record showing the failure, diagnosis and repaired state.

Break / fix · diagnose

Inspect the intermediate evidence before changing the model or prompt.
If a command fails, preserve the error and classify which pipeline layer failed.
Change one pipeline variable at a time and rerun the same test case.
Do not accept a plausible answer as proof; verify the intermediate evidence and source identity.

Production view

Production RAG improves when every change can be traced to evidence at a specific pipeline layer. In production, retain the before/after evidence so the same failure can become a regression test.

Production checklist

What would an engineer verify before shipping this layer?

Gate 1

Critical gates defined

Gate 2

HOLD path exercised

Gate 3

Evidence dossier and rollback/handover complete

Avoid these traps

Common beginner and engineering mistakes

Promoting because the demo looks good.
Reporting one aggregate quality score without layer diagnostics.
Shipping without deletion/freshness procedures.

Knowledge check

Can you explain the idea and diagnose the system?

1What makes the capstone production-shaped?

It integrates retrieval, evidence, evaluation, security, lifecycle and operations rather than only answer generation.

2What should block promotion?

Critical failures in authorization, grounding, security or required acceptance criteria.

3Why retain an evidence dossier?

For reproducibility, review, regression analysis and operational handover.

4What is the final learner outcome?

The ability to design, evaluate, troubleshoot and operate a trustworthy RAG/knowledge system.

My Learning

Record your progress for this module.

Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.

Not started

Core lab attemptedRecord this after meaningfully completing or attempting the module's core lab.Knowledge checks attemptedThis is marked automatically after you open every answer in the module; you can also record it manually.

Attempt the core lab and knowledge checks before marking the module complete.