Retrieval

Level 2 builds every box of that pipeline from nothing, starting in a library.

The everyday picture. You walk into a library with a question. Two librarians are on duty.

• The keyword librarian takes your words literally. They count how often each word of your question appears in each book, give rare words far more weight than common ones, stop getting more excited after the tenth mention, and are a little suspicious of very long books that mention everything. Ask about "ERR-4012" and they find the one page that says "ERR-4012". Ask about "automobile reimbursement" and they find nothing, because the book says "car" and "reimbursed".
• The meaning librarian understands what you mean: "automobile" is a "car", "scam" is "phishing". But they remember the gist of each book, not its serial numbers, so "ERR-4012" and "ERR-4013" blur together.

Real systems ask both and trust the books both recommend. Then an expert reads the top handful of candidates carefully, next to your question, and puts them in final order. That's the whole modern retrieval pipeline, and this lesson builds every piece of it.

Words used throughout, in plain terms:

• Retrieval. Finding the documents that answer a question, ranked best first. In a RAG system (primer.agents.rag) the language model can only use what retrieval hands it, so retrieval quality caps answer quality.
• Token. Here, one word after lowercasing and dropping filler words ("the", "is"): see primer.common.text.
• Recall@k. The share of questions whose answer appears in the top k results. MRR (mean reciprocal rank) also rewards putting the answer first: an answer at rank 1 scores 1, at rank 2 scores 1/2, missing scores 0, averaged over questions.

The full pipeline, which the sections below build one box at a time:

flowchart LR
  Q[Question] --> K[Keyword search<br/>BM25]
  Q --> D[Meaning search<br/>embeddings]
  K --> F[Fuse the two rankings<br/>RRF]
  D --> F
  F --> S[Shortlist<br/>top 10 to 100]
  S --> R[Rerank each candidate<br/>with the question: cross-encoder]
  R --> T[Top 3 to 5 passages<br/>to the language model]

The everyday picture. The keyword librarian from above: rare words count more, repetition helps less and less, and long books get a small penalty for mentioning everything.

A tiny worked example. Three one-line "documents":

The average length is (3 + 2 + 1)/3 = 2. Search for cat. It appears in one of the three documents, so it's rare and gets a high weight (0.981, computed below); "dog" is in two, so it would get less (0.470). Only d₁ contains "cat", twice, and d₁ is a bit longer than average. Plugging in, d₁ scores 1.207 and the others score 0.

flowchart LR
  Q[Query words] --> IDF["Rarity weight per word<br/>IDF: rarer = heavier"]
  Q --> TF["Count in this document<br/>tf, with saturation"]
  DL["Document length vs.<br/>the average length"] --> TF
  IDF --> M((×))
  TF --> M
  M --> SUM["Add up over<br/>the query's words"]
  SUM --> S[BM25 score]

On the example (query "cat", document d₁): N = 3, n(cat) = 1, so IDF = ln(1 + 2.5/1.5) = ln(2.667) = 0.981. f = 2, |d| = 3, avgdl = 2: the denominator is 2 + 1.5 · (1 − 0.75 + 0.75 · 3/2) = 2 + 1.5 · 1.375 = 4.0625. The count part is 2 · 2.5 / 4.0625 = 1.231. Score = 0.981 · 1.231 = 1.207.

In Python:

import math
# 3 documents; 1 of them contains "cat"
N, n_t = 3, 1
# ln(1 + (N - n(t) + 0.5) / (n(t) + 0.5))
IDF = math.log(1 + (N - n_t + 0.5) / (n_t + 0.5))
round(IDF, 3)  # → 0.981
# f(cat, d1), |d1|, average document length
f, d_len, avgdl = 2, 3, 2
k_1, b = 1.5, 0.75
count_part = f * (k_1 + 1) / (f + k_1 * (1 - b + b * d_len / avgdl))
round(count_part, 3)  # → 1.231
# Σ over the query's only word
round(IDF * count_part, 3)  # → 1.207

In code: BM25 precomputes each word's IDF and each document's length; BM25.scores applies the formula to every document, and BM25.search returns the top k with a non-zero score.

Why it matters BM25 is decades old, needs no training, runs on any search engine (Elasticsearch, OpenSearch, Lucene), and is still very hard to beat on exact identifiers: product codes, error numbers, names, ticket IDs. Those are everywhere in company data.

The everyday picture. The meaning librarian writes a one-line summary card for every book before anyone asks anything. When your question arrives, they write a summary card for it too and pull the books whose cards are most similar. "Automobile reimbursement" and "car mileage expense" get similar cards even though they share no words.

A tiny worked example. With this repo's toy embedding model (primer.common.embedder), the query "automobile reimbursement" and the car-mileage article share zero words, yet their vectors have cosine similarity of about 0.8, the highest in the collection. "What does ERR-4012 mean", by contrast, puts the right article only 4th: the code is one blurred word among many, and "what" and "mean" pull the vector elsewhere.

flowchart LR
  subgraph AHEAD["Ahead of time"]
    D[Each document] --> E1[Encoder] --> V[(Document vectors)]
  end
  subgraph NOW["At question time"]
    Q[Question] --> E2[Same encoder] --> QV[Question vector]
  end
  QV --> NN["Nearest vectors<br/>(ann.py)"]
  V --> NN
  NN --> R[Ranked documents]

In code: SearchEngine embeds every document once with primer.common.embedder.ConceptEmbedder into a primer.ml.embeddings.ann.FlatIndex; SearchEngine.dense embeds the question and returns the nearest documents. doc_text decides what gets indexed: the title plus the body.

Why it matters dense retrieval handles paraphrases, synonyms and other languages, but it blurs exact tokens. Measure both kinds of queries on your own data before trusting either librarian alone.

The everyday picture. Ask both librarians for their top-10 lists, then hold a vote. A book earns points for each list it's on, more the higher it sits. Books both librarians rank highly rise to the top. Crucially, you only compare positions, never the librarians' private scoring systems, which aren't on the same scale.

A tiny worked example. With the standard constant k = 60:

X, which both methods like, beats Y, which only one method loves.

flowchart LR
  Q[Question] --> B[BM25 ranking]
  Q --> D[Dense ranking]
  B --> R["For every document:<br/>add 1/(60 + rank)<br/>from each list it's on"]
  D --> R
  R --> O[Sort by total:<br/>the fused ranking]

On the example: X: 1/(60+1) + 1/(60+3) = 0.01639 + 0.01587 = 0.0323. Y: 1/(60+1) = 0.0164.

In Python:

k = 60
# d's position in each ranking that contains it
def RRF(ranks):
    # Σ_i 1/(k + rank_i(d))
    return sum(1 / (k + rank_i) for rank_i in ranks)
# X: 1st in one ranking, 3rd in the other
round(RRF([1, 3]), 4)  # → 0.0323
# Y: on one ranking only
round(RRF([1]), 4)  # → 0.0164

In code: reciprocal_rank_fusion adds up 1/(k + rank) across any number of rankings; SearchEngine.hybrid fuses SearchEngine.bm25 with SearchEngine.dense, and evaluate measures recall@k and MRR for any search method.

Why it matters company data is full of both paraphrase-style questions and exact identifiers, so hybrid search almost always beats either method alone. It's cheap to add: most search engines and vector databases support it built in.

The everyday picture. A matchmaker has two ways to work. The fast way: write an index card for each person once, then match cards, so thousands of people can be pre-filed. The careful way: sit the two people down together and watch them talk. That's far more accurate, but every pair needs its own meeting. A bi-encoder is the index card. A cross-encoder is the meeting. So you use cards to shortlist, then hold meetings only with the shortlist.

A tiny worked example: a hard negative. "What are the password rules?" has two candidate answers on the same topic. The password reset how-to (it-001) mentions "password" three times; the password policy (it-002) is the real answer. BM25 and the fused hybrid ranking both put the how-to first. A hard negative is exactly this: a document that looks relevant (right topic) but doesn't answer the question.

Our toy cross-encoder reads the question and the document together and checks which of the question's ideas the document covers:

The score weighs the features: coverage + 0.5 × phrase + 0.25 × exact + 0.25 × cosine, where the cosine is the ordinary bi-encoder similarity (0.63 for the policy, 0.27 for the how-to). For the policy that's 1.0 + 0.5 + 0.125 + 0.16 ≈ 1.78; for the how-to, 0.5 + 0 + 0.125 + 0.07 ≈ 0.69. After reranking, the policy is first.

flowchart TB
  subgraph BI["Bi-encoder: two separate encodings"]
    direction LR
    Q1[Question] --> EQ[Encoder] --> VQ[vector]
    D1[Document] --> ED[Encoder] --> VD[vector]
    VQ --> COS[cosine]
    VD --> COS
  end
  subgraph CROSS["Cross-encoder: one joint reading"]
    direction LR
    QD["[Question] + [Document]<br/>as one input"] --> J["Encoder with attention<br/>across both texts"] --> SC[relevance score]
  end

The cost argument. Say a cross-encoder pass takes 10 ms on a GPU. Over a 1-million-document collection that's 10,000 seconds per question. Over a shortlist of 50, it's 0.5 s, or much less when the pairs are batched. That is why the standard pipeline is retrieve wide and cheap, then rerank narrow and precise, and why you cap the shortlist to keep latency predictable.

In code: CrossEncoder.score reads one question-document pair and adds CrossEncoder.coverage, CrossEncoder.phrase and CrossEncoder.exact, with ideas grouping synonyms into ideas. retrieve_then_rerank shortlists with hybrid search, then reorders the shortlist with the cross-encoder.

Why it matters, and a warning: rerankers are models too, and can be wrong. On this repo's 20 questions, reranking fixes the hard negatives but demotes one answer ("refund for the hotel" → it prefers the car-mileage article, which talks about reimbursing trips). Measure recall@k and MRR with and without the reranker before shipping it.

The everyday picture. Instead of one summary card per book, keep a card for every word in the book. When your question arrives, each of your words looks for its single best-matching card in the book, and the book's score is how well your words found partners. That's more precise than one summary per book, and unlike the matchmaker's meetings, the cards can still be written ahead of time.

A tiny worked example. Two-number word vectors. The query has words (1, 0) and (0, 1); the document has words (1, 0) and (0.6, 0.8).

Score = 1.0 + 0.8 = 1.8.

flowchart LR
  QT["Query: one vector<br/>per word"] --> SIM["Similarity of every<br/>query word × doc word"]
  DT["Document: one vector<br/>per word (precomputed)"] --> SIM
  SIM --> MAX["Each query word keeps<br/>its best match (max)"]
  MAX --> SUM["Add the maxima"] --> S[Score]

On the example: max(1.0, 0.6) + max(0.0, 0.8) = 1.0 + 0.8 = 1.8.

In Python:

# one vector per query word
q = [(1, 0), (0, 1)]
# one vector per document word
d = [(1, 0), (0.6, 0.8)]
def dot(a, b):
    return sum(a_k * b_k for a_k, b_k in zip(a, b))
# each query word's best match
[max(dot(q_i, d_j) for d_j in d) for q_i in q]  # → [1, 0.8]
# Σ_i max_j q_i · d_j
sum(max(dot(q_i, d_j) for d_j in d) for q_i in q)  # → 1.8

In code: maxsim computes the score from two sets of word vectors; LateInteraction.token_vectors gives a text one vector per word, and LateInteraction.search ranks documents by MaxSim.

Why it matters late interaction gets much of a cross-encoder's precision while keeping precomputed documents. The cost is storage: one vector per word instead of per document, often 50 to 200 times more. ColBERTv2 compresses those vectors to make it practical.

The everyday picture. You're turning a cookbook into index cards. Cut every 50 words and a recipe's title lands on one card and its oven temperature on the next. Nobody searching for that recipe finds the temperature. Cut at each recipe instead, and every card is self-contained.

A tiny worked example. A 130-word text, 50-word chunks with 10 words of overlap: windows start every 50 − 10 = 40 words, at 0, 40 and 80, giving 3 chunks (words 1–50, 41–90, 81–130). Each shares 10 words with the next, so a sentence cut at a boundary survives whole in at least one chunk, if it's short.

On this lesson's remote-work handbook, the question "home internet stipend" shows the difference. With 40-word fixed chunks, the best-matching chunk holds the heading "Home internet stipend" but the amount, "50 dollars per month", fell into the next window. With structure-aware chunks, cut at headings, the best chunk holds both.

flowchart LR
  DOC[Document] --> P[Parse: headings,<br/>paragraphs, tables]
  P --> SPLIT["Split at structure;<br/>merge small pieces<br/>up to a size limit"]
  SPLIT --> CTX["Prefix each chunk with<br/>its heading (context)"]
  CTX --> META["Attach metadata: source,<br/>section, date, access list"]
  META --> EMB[Embed + index]

Parent-child retrieval. Small chunks match precisely; big chunks give the model enough context. Get both: index small children (single sentences), and when one matches, hand the model its parent (the whole section).

flowchart LR
  Q[Question] --> C["Search small children<br/>(sentences)"]
  C --> HIT[Best child]
  HIT --> PARENT["Look up its parent<br/>(the whole section)"]
  PARENT --> LLM[Give the parent<br/>to the model]

On the example: ⌈(130 − 10)/(50 − 10)⌉ = ⌈120/40⌉ = 3.

In Python:

import math
# words, chunk size, overlap
n, s, o = 130, 50, 10
# ⌈(n - o) / (s - o)⌉: the step is s - o
math.ceil((n - o) / (s - o))  # → 3

In code: fixed_size_chunks cuts overlapping windows of words; structure_aware_chunks cuts at headings and paragraphs and returns Chunk records that carry their heading and metadata. best_chunk picks the chunk BM25 ranks highest, and parent_child_search matches a sentence and returns its whole section. chunk_engine indexes one document's chunks so that retrieve_then_rerank runs on them, as in the widget above.

Why it matters chunking choices often matter more than the choice of embedding model. Split on structure, keep headings with their content, add modest overlap, and attach metadata for filtering.

The everyday picture. A filing clerk was trained on sheets stamped "QUESTION:" or "DOCUMENT:", and files each kind in a matching drawer. Hand them an unstamped sheet and they don't complain: they file it anyway, in a drawer nobody will look in. Nobody notices until people stop finding things.

Questions are short and phrased as asks; passages are long statements. Some embedding models (the E5 family, for example) are trained to expect a "query: " prefix on questions and a "passage: " prefix on documents, so they can encode each side appropriately. Forget a prefix and every vector still looks normal (right length, plausible scores) but sits in a slightly wrong part of the space.

A tiny worked example. This lesson simulates such a model. With both prefixes, 11 of the 12 everyday questions find their answer in the top 3. Index the documents without "passage: " and the same questions fall to 9 of 12, and MRR drops from 0.875 to about 0.57. No error is raised anywhere.

flowchart LR
  subgraph OK["Correct"]
    Q1["'query: ' + question"] --> E1[Model] --> A1[aligned vectors]
    D1["'passage: ' + document"] --> E1
  end
  subgraph BUG["Prefix forgotten at indexing"]
    Q2["'query: ' + question"] --> E2[Model] --> A2[question vectors]
    D2["document (no prefix)"] --> E2 --> B2[drifted document vectors]
  end

In code: PrefixedEmbedder simulates such a model: PrefixedEmbedder.encode_one strips a known prefix and encodes normally, but partly rotates the vector of any text that lacks one.

Why it matters always read the embedding model's card for required prefixes or instructions, use the same model and settings at indexing and query time, and keep a small labeled set so silent regressions show up.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.