Retrieval-augmented generation (RAG), end to end

What follows builds every box of the pipeline from scratch, in the order a question travels through them, then the upgrades, then how to tell a retrieval failure from a generation failure.

An open-book exam with a librarian. Before you answer, the librarian fetches the few pages most likely to contain the answer; you answer from those pages and write down which page each fact came from. If the pages don't contain the answer, you say so instead of guessing.

Retrieval-augmented generation is that arrangement for a language model. The model's training knowledge is frozen and doesn't include your company's documents, so before each answer the system retrieves relevant passages from your documents and puts them in the prompt, and the model generates an answer from them, with citations. It's how you give a model knowledge that changes, or that must be cited, without retraining it.

flowchart LR
  subgraph ING["Ingestion, ahead of time"]
    D[Documents] --> P[Parse] --> C[Chunk + metadata] --> E[Embed + keyword index]
  end
  E --> IX[(Index)]
  subgraph QRY["Query time, per question"]
    Q[Question] --> F[Filter by permissions<br/>and date] --> R[Retrieve: hybrid] --> RR[Rerank] --> AC[Assemble context] --> G[Generate + cite] --> V[Verify citations]
  end
  IX --> F

In code: RAGIndex is the top row: it chunks every document and builds the keyword and dense indexes. answer is the bottom row, from filtered retrieval to verified citations, and returns a RAGResult.

The rest of this lesson walks the boxes in order, then covers the upgrades and how to debug a wrong answer.

Everyday picture Photocopy a newspaper page and read it straight across: you get the first line of one column, then the first line of the next story, and nothing makes sense. Text extracted from PDFs has the same problems: running headers and page numbers mixed into the text, words broken across lines, and tables flattened into a row of numbers with no columns.

Worked example MESSY_PDF is two pages of a travel policy as a naive extractor returns it. clean_extracted_text repairs it in four steps:

The table is the dangerous one. Flattened, "200" floats free of its grade. table_rows_as_sentences turns each row into a self-contained sentence, Grade L4-L6: hotel cap 200, meal cap 75., so any row can be retrieved on its own and still says what its numbers mean.

flowchart LR
  RAW[Raw extracted text] --> H[Drop lines repeated<br/>on most pages]
  H --> N[Drop page numbers]
  N --> J[Rejoin hyphenated words,<br/>unwrap paragraphs]
  J --> T{Table rows?}
  T -->|yes| S[One sentence per row,<br/>headers repeated]
  T -->|no| P[Paragraphs]
  S --> OUT[Clean text for chunking]
  P --> OUT

Why it matters Enterprise documents are scanned, multi-column, full of tables that span pages. If parsing garbles the numbers, no retriever or model can recover them, and the failure looks like a model error.

In code: extract_table finds a table by its header row in the cleaned text and returns each row as a dict, ready for table_rows_as_sentences.

Everyday picture Index cards. Each card holds one idea, small enough to match a question precisely, and in its corner a label: which document, which date, who may read it.

Worked example With a 15-word limit, it-004 splits into two passages:

(The limit is tiny because the toy documents are tiny; real systems use a few hundred tokens.) Split on the document's structure (sentences, paragraphs, sections), not at fixed character counts that cut a sentence in half. primer.ml.embeddings.retrieval compares chunking strategies in depth.

Why it matters Small chunks match precisely but can lose context (the second card doesn't say which error it fixes; see contextual retrieval below). The metadata is what makes permission and date filters possible later.

In code: chunk_document splits a document on sentence boundaries into Passages, and each Passage carries its id, title, department, date and readers.

Everyday picture Two librarians: one searches the catalogue for your exact words, the other understands what you mean even when you use different words. You take the books both of them rank highly.

• Keyword search (BM25) scores passages by the question's words, giving more weight to rare words and less to long passages. It nails exact identifiers like ERR-4012 and misses synonyms.
• Dense search compares embeddings (vectors where closeness means similar meaning; see primer.ml.embeddings). It finds "scam message in my inbox" → the phishing guide with no shared words, but blurs exact codes.
• Hybrid search runs both and merges the two rankings with reciprocal rank fusion (RRF), which only needs the ranks, never the scores. BM25 scores and cosine similarities live on different scales, so adding them would be meaningless.

k = 60
# 1st by dense, 3rd by keyword: Σ_i 1/(k + rank_i(d))
round(1 / (k + 1) + 1 / (k + 3), 4)  # → 0.0323
# 1st in one list, absent from the other
round(1 / (k + 1), 4)  # → 0.0164

# 𝟙[...] for each of the 12 questions; ERR-4012 is the 0
found = [1] * 11 + [0]
# (1/|Q|) Σ over q in Q
round(sum(found) / len(found), 2)  # → 0.92

In code: RAGIndex.retrieve ranks the allowed passages with primer.ml.embeddings.retrieval.BM25, with dense similarity, or with both fused by primer.ml.embeddings.retrieval.reciprocal_rank_fusion, depending on the mode you ask for. recall_curve measures recall@k for each method.

Everyday picture The librarians bring back twenty books quickly; an expert then reads your question next to each book and picks the best three.

A reranker (usually a cross-encoder: a model that reads the question and one passage together and outputs a relevance score) is far more precise than comparing precomputed vectors, and far too slow to run over every passage. So retrieval casts a wide, cheap net (here 8 candidates) and the reranker keeps the best few (here 3). This module reuses the toy cross-encoder from primer.ml.embeddings.retrieval.

In code: RAGIndex.rerank scores each (question, passage) pair with primer.ml.embeddings.retrieval.CrossEncoder and keeps the best few.

Worked example "What does ERR-4012 mean?" becomes this context (each source escaped and tagged with its id; the question last; see primer.agents.context):

<sources>
<source id="it-004#1" title="Error ERR-4012: VPN tunnel failed" updated="2026-04-10">Update AnyConnect to version 5.1 or later and reboot.</source>
<source id="it-004#0" title="Error ERR-4012: VPN tunnel failed" updated="2026-04-10">ERR-4012 means the VPN tunnel could not be established, usually because the client is out of date.</source>
<source id="it-002#2" title="Password policy" updated="2025-11-20">This policy applies to all employees and contractors.</source>
</sources>
<question>What does ERR-4012 mean?</question>

and the answer is "ERR-4012 means the VPN tunnel could not be established, usually because the client is out of date. [it-004#0]". verify_citations then checks that every cited id was really in the context and that the cited passage supports the claim before it; a question the sources don't cover gets "I don't know based on the provided sources."

sequenceDiagram
  participant U as User
  participant A as RAG service
  participant I as Index
  participant M as Model
  U->>A: What does ERR-4012 mean?
  A->>I: hybrid search, filtered to what this user may read
  I-->>A: 8 candidate passages
  A->>A: rerank, keep 3
  A->>M: rules + tagged sources + question
  M-->>A: answer with [it-004#0]
  A->>A: verify: id was in context, passage supports claim
  A-->>U: answer + clickable citation

In code: assemble_context builds the tagged sources and the question. generate sends them to the model with the answer-from-sources rules, and grounded_policy is the offline stand-in model that honours them. answer chains retrieve, rerank, assemble, generate and verify_citations.

Everyday picture A librarian checks your library card before fetching from the restricted archive. Checking it after you've read the document is pointless.

Each passage carries its access-control list (ACL: the groups allowed to read it), copied from the source system (SharePoint, Google Drive, a wiki). RAGIndex.retrieve drops every passage the user can't read before ranking, so restricted text can't reach the context, the model, or the answer.

Worked example "What is the Q3 revenue forecast?" (the forecast document is readable only by finance and exec):

sequenceDiagram
  participant U as Employee (not finance)
  participant A as RAG service
  participant M as Model
  rect rgb(250, 225, 225)
  Note over A,M: Wrong: filter after generation
  A->>M: sources include the restricted forecast
  M-->>A: "…41 million dollars [fin-006#0]"
  A->>A: strip the restricted citation
  A-->>U: "…41 million dollars" (leaked)
  end
  rect rgb(225, 240, 225)
  Note over A,M: Right: filter before retrieval
  A->>A: drop passages the user can't read
  A->>M: sources without the forecast
  M-->>A: "I don't know…"
  A-->>U: nothing restricted
  end

In code: post_generation_filter_answer is the wrong way, kept so the leak can be shown: it retrieves for every group, generates, then strips the restricted citation markers.

Each upgrade below is a working function, and each fixes a failure you can reproduce in demo():

HyDE (Hypothetical Document Embeddings) deserves a picture, because it sounds backwards: you ask a model to invent an answer, then search with it.

flowchart LR
  Q["Question:<br/>'weird message asking<br/>for my bank details'"] --> M[Model drafts a<br/>plausible answer]
  M --> H["Draft: 'This sounds like a phishing<br/>email… report it in Outlook'"]
  H --> E[Embed the draft]
  E --> S[Dense search]
  S --> R[Phishing guide, rank 1]

Contextual retrieval fixes the "second index card" problem from the chunking section: before indexing, each passage gets a short line saying where it comes from. Here that's just the title and department; Anthropic's version asks a model to write one sentence situating the chunk in its document.

Everyday picture A researcher rather than a librarian: reads the question, decides what to look up, reads the results, and searches again if something's missing.

Plain RAG searches exactly once, with the user's words. In agentic RAG search is a tool the model calls: zero times for small talk, several times for a multi-part question, with its own reformulated queries.

Worked example "What is the per-diem for meals and how do I upload expense receipts?" produces two searches ("What is the per-diem for meals", "how do I upload expense receipts") and an answer citing both fin-002#2 and fin-004#0. "thanks, that's all!" produces no search at all.

sequenceDiagram
  participant U as User
  participant M as Model
  participant S as search_kb tool
  U->>M: per-diem for meals AND how to upload receipts?
  M->>S: search_kb("per-diem for meals")
  S-->>M: fin-002 passages
  M->>S: search_kb("how do I upload expense receipts")
  S-->>M: fin-004 passages
  M-->>U: both answers, each with its citation

In code: agentic_answer offers the model one search tool and runs the loop: each search retrieves, reranks and returns tagged passages, until the model answers or the step limit is hit.

Everyday picture A detective's corkboard: photos of people and places joined by string, each string labelled with the note that established it.

Some questions aren't answered by any single passage: "what depends on two-factor authentication?" or "what are the main themes across these contracts?". GraphRAG first extracts entities (named things: systems, people, products) and the relationships between them into a graph, then answers by walking it, or by summarizing groups of connected entities. build_graph does the smallest honest version: two entities are linked when one sentence mentions both, and each link remembers which document said so.

flowchart LR
  TF((two-factor)) ---|it-006| VPN((VPN))
  TF ---|it-006| EM((email))
  TF ---|it-006| AA((authenticator app))
  TF ---|it-003| AC((AnyConnect))
  TF ---|it-003| SC((software center))
  TF ---|hr-003| LT((laptop))
  AC ---|it-003| SC
  VPN ---|it-006| EM
  AA ---|it-001| SSP((self-service portal))

In code: communities groups the graph from build_graph into connected clusters of entities, the "themes" a real GraphRAG system would summarize.

Everyday picture A student gets an exam question wrong. Either the librarian brought the wrong book, or the student misread the right one. The fixes are completely different, so find out which before changing anything.

Worked example debug_wrong_answers runs the 12 labelled questions and classifies each: retrieval miss (no relevant document in the top 3: no prompt can fix it), generation miss (the right document was there but the answer didn't use it), or ok. With dense-only retrieval, "what does ERR-4012 mean" is a retrieval miss and recall@3 is 0.92; switching to hybrid fixes it (recall@3 = 1.00). What's left are generation misses. One of them, "per-diem for meals when traveling", is answered from the superseded 2023 policy: a data problem that a date filter fixes. The other, "enroll in MFA", is a limit of this lesson's stand-in model rather than of RAG: it matches words literally, so "enroll" misses the passage's "enrolling", and with only "MFA" in common it declines to answer. A real model would read it-006 and answer.

flowchart TD
  W[Confident wrong answer] --> R{Right passage in<br/>the top k?}
  R -->|no| P{Is it in the<br/>index at all?}
  P -->|no| FIXI[Fix ingestion:<br/>parsing, chunking, permissions sync]
  P -->|yes| FIXR[Fix retrieval:<br/>hybrid, rerank, rewrite, HyDE, filters]
  R -->|yes| C{Was it in the<br/>final context?}
  C -->|no| FIXC[Fix assembly:<br/>reranker, budget, top_n]
  C -->|yes| FIXG[Fix generation:<br/>instructions, citations, stale sources]

Answer each one out loud or on paper before you open it. If you can explain it, you know it.