Retrieval Pipelines: Chunking, Recall, Reranking, and Citations
A document-answering system must find supporting text before asking a model to answer from it. When the evidence never reaches the context, a larger generator may simply produce a more fluent guess. This page follows documents into evidence; agent memory addresses what to preserve across tasks.
Two cases need no retrieval at all. If the material fits in the context window as a whole, put it there. If you are looking for a known function name or error in a code base, exact and symbol search beat semantic retrieval (see context engineering).
Choose chunks from the question backward
Consider a fictional equipment manual. One section describes battery replacement; the next says that waterproofing assumes an intact enclosure. A user asks, “Can I immerse it immediately after replacing the battery?” Finding only a paragraph containing “battery” misses the decisive condition.
A chunk should preserve a complete thought without burying evidence in unrelated text. Start from headings, paragraphs, functions, or table boundaries, then cap unusually long sections. Overlap can preserve boundary sentences but also creates duplicate candidates. API reference entries, chat logs, and research chapters need different boundaries; one fixed chunk length cannot serve them all.
Keep the source identifier, location, heading path, and version with every chunk. A retrieved table row must retain enough information to recover its headings and units. Detached text makes citation, deduplication, and updates harder. Deleted documents must invalidate their old index entries as well.
Separate retrieval from reranking
Retrieval selects plausible passages from the corpus; reranking examines how each candidate relates to the question. The Sentence Transformers example illustrates lexical or bi-encoder retrieval followed by cross-encoder reranking.
Hybrid search can combine lexical and vector candidates, but their score scales need not match. A simple baseline is to combine ranks rather than directly add cosine similarity and lexical scores: reciprocal rank fusion adds 1 / (c + rank) per candidate across lists, with c controlling the advantage of early positions. It is a ranking rule, not a probability of correctness.
Locate a failure before changing the model
For the battery question, suppose retrieval keeps 20 passages, reranking selects 4, and the context budget permits 2. These are illustrative budgets, not recommended defaults.
If the waterproofing condition ranks 30th initially, retrieval failed. If it entered the pool but reranking pushed it to 12th, ranking failed. If it reached the top 4 but duplicates displaced it, assembly failed. If it entered the prompt and the answer ignored it, generation failed. Keeping candidate IDs and positions at each stage makes these failures distinguishable. The final answer alone cannot tell you whether to change embeddings, chunking, or instructions.
For questions spanning sections, retrieve a small passage and expand to adjacent paragraphs or its parent section. Expansion also brings irrelevant text, so budget it and preserve the conditions the question needs. Apply access, time, and version constraints during candidate selection rather than discovering after generation that the evidence was inapplicable.
Attach citations to claims
“See the manual” is insufficient. A claim such as “the seal must be checked after replacement” should point to the passage supporting it, not merely the whole document. If versions disagree, resolve their applicable dates and equipment models. Preserve a remaining conflict instead of having the model invent a unified conclusion.
Retrieved text is evidence, not an execution instruction. A passage requesting that the agent ignore rules or invoke tools gains no authority from a high rank. Tool contracts explain that boundary.
A score is only a ranking signal, not a fact. Open the source behind the top passage and check it. No result means only that this chunking, tokenisation and model did not surface the text; it does not prove the corpus lacks it. Before changing code or writing a conclusion, reopen the current source, because the index reflects the files as they were when it was built.
A minimal runnable baseline
Build the smallest version that runs before deciding whether you need a vector database, a reranker or cleverer chunking. The Python below splits a folder by Markdown heading and runs BM25 lexical recall and vector recall side by side. Reciprocal rank fusion merges the two lists and returns the top 3 passages as file:line. Finally it scores a labelled question list. It needs two libraries: rank_bm25 for BM25 and Model2Vec to load the local potion-multilingual-128M. That model is distilled from BGE-M3 and handles Chinese and English; the first run downloads about 0.5 GB, and after that it is fast even on a CPU.
import pathlib, re, sys
import numpy as np
from model2vec import StaticModel
from rank_bm25 import BM25Okapi
def chunks(folder): # one chunk per Markdown heading, remembering file and line
for path in sorted(pathlib.Path(folder).rglob("*.md")):
lines = path.read_text(encoding="utf-8").splitlines()
heads = [i for i, line in enumerate(lines) if line.startswith("#")] + [len(lines)]
for a, b in zip(heads, heads[1:]):
if "\n".join(lines[a + 1:b]).strip(): # skip headings with nothing under them
yield {"where": f"{path}:{a + 1}", "text": "\n".join(lines[a:b])}
def tokens(text): # English words and single Chinese characters
return re.findall(r"[a-z0-9]+|[\u4e00-\u9fff]", text.lower())
docs = list(chunks(sys.argv[1]))
bm25 = BM25Okapi([tokens(d["text"]) for d in docs])
model = StaticModel.from_pretrained("minishlab/potion-multilingual-128M")
vectors = model.encode([d["text"] for d in docs])
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
def search(question, k=3, pool=20, c=60):
lexical = np.argsort(-bm25.get_scores(tokens(question)))[:pool]
q = model.encode([question])[0]
semantic = np.argsort(-(vectors @ (q / np.linalg.norm(q))))[:pool]
fused = {}
for ranking in (lexical, semantic): # reciprocal rank fusion
for rank, i in enumerate(ranking, start=1):
fused[i] = fused.get(i, 0) + 1 / (c + rank)
return [docs[i]["where"] for i in sorted(fused, key=fused.get, reverse=True)[:k]]
rows = [line.rstrip("\n").split("\t") for line in open(sys.argv[2], encoding="utf-8")]
hits = 0
for question, expected in rows:
found = search(question)
hits += expected in found
print("ok " if expected in found else "MISS", question, "->", ", ".join(found))
print(f"recall@3: {hits}/{len(rows)}")
The question list is a tab-separated file: one question per line, followed by the file:line of the heading its answer sits under.
换完电池还能直接浸水吗 notes/manual.md:7
How do I update the firmware? notes/faq.md:7
Save the code as rag.py, put the notes in a notes/ folder, and run:
pip install rank_bm25 model2vec
python rag.py notes questions.tsv
With two notes written from the device manual above and 10 questions, the script found 10 of 10 in a test on 2026-10-29; each output line shows a question and the three locations it found. BM25 leaves tokenisation to you: the code splits English into words and Chinese into single characters, which is enough for a small corpus; a larger Chinese corpus deserves a word segmenter. The vectors sit in memory and are compared one by one, so tens of thousands of passages need no vector index.
The question list is worth more than the code. Rerun it after every chunking or model change and look for questions that stopped hitting. For choosing a ready-made tool, see the comparison table in agent memory. Continue with retrieval and generation evaluation for metrics and counterexamples, and context engineering for allocating the model's working space.