Retrieval Pipelines: Chunking, Recall, Reranking, and Citations
A document-answering system must find supporting text before asking a model to answer from it. When the evidence never reaches the context, a larger generator may simply produce a more fluent guess. This page follows documents into evidence; agent memory addresses what to preserve across tasks.
Choose chunks from the question backward
Consider a fictional equipment manual. One section describes battery replacement; the next says that waterproofing assumes an intact enclosure. A user asks, “Can I immerse it immediately after replacing the battery?” Finding only a paragraph containing “battery” misses the decisive condition.
A chunk should preserve a complete thought without burying evidence in unrelated text. Start from headings, paragraphs, functions, or table boundaries, then cap unusually long sections. Overlap can preserve boundary sentences but also creates duplicate candidates. API reference entries, chat logs, and research chapters need different boundaries; one fixed chunk length cannot serve them all.
Keep the source identifier, location, heading path, and version with every chunk. A retrieved table row must retain enough information to recover its headings and units. Detached text makes citation, deduplication, and updates harder. Deleted documents must invalidate their old index entries as well.
Separate retrieval from reranking
Retrieval selects plausible passages from the corpus; reranking examines how each candidate relates to the question. The Sentence Transformers example illustrates lexical or bi-encoder retrieval followed by cross-encoder reranking.
Hybrid search can combine lexical and vector candidates, but their score scales need not match. Rank fusion is a simple baseline: reciprocal rank fusion adds 1 / (c + rank) per candidate across lists, with c controlling the advantage of early positions. It is a ranking rule, not a probability of correctness.
Locate a failure before changing the model
For the battery question, suppose retrieval keeps 20 passages, reranking selects 4, and the context budget permits 2. These are illustrative budgets, not recommended defaults.
If the waterproofing condition ranks 30th initially, retrieval failed. If it entered the pool but reranking pushed it to 12th, ranking failed. If it reached the top 4 but duplicates displaced it, assembly failed. If it entered the prompt and the answer ignored it, generation failed. Keeping candidate IDs and positions at each stage makes these failures distinguishable. The final answer alone cannot tell you whether to change embeddings, chunking, or instructions.
For questions spanning sections, retrieve a small passage and expand to adjacent paragraphs or its parent section. Expansion also brings irrelevant text, so budget it and preserve the conditions the question needs. Apply access, time, and version constraints during candidate selection rather than discovering after generation that the evidence was inapplicable.
Attach citations to claims
“See the manual” is insufficient. A claim such as “the seal must be checked after replacement” should point to the passage supporting it, not merely the whole document. If versions disagree, resolve their applicable dates and equipment models. Preserve a remaining conflict instead of having the model invent a unified conclusion.
Retrieved text is evidence, not an execution instruction. A passage requesting that the agent ignore rules or invoke tools gains no authority from a high rank. Tool contracts explain that boundary.
Place tools along the pipeline
QMD searches Markdown notes; zvec-grep supports semantic search in workspace code and documents; Basic Memory exposes persistent notes and relationships. Their storage and query mechanisms need not share one database. Their return contracts should identify the source, location, version, useful passage, and applicable access scope.
Start with a small set of questions that have known source evidence. Check separately whether evidence was retrieved, ranked highly, and used correctly. Continue with retrieval and generation evaluation for metrics and counterexamples, and context engineering for allocating the model's working space.