Retrieval That Actually Retrieves
Why naive vector search disappoints: chunking that destroys meaning, embeddings that miss exact terms, and no reranking. Hybrid retrieval, chunk design, and measuring recall before you blame the model.
Why naive vector search disappoints: chunking that destroys meaning, embeddings that miss exact terms, and no reranking. Hybrid retrieval, chunk design, and measuring recall before you blame the model.
Your first retrieval demo went beautifully. You split the support handbook into chunks, embedded them, asked "how do I reset the controller", and the right paragraph came back at rank one. You shipped it that afternoon.
Then a support engineer typed "what does E-412 mean" and got three
passages about error handling in general, not one of which
contained the string E-412. The model wrote a confident
paragraph about cable seating, because that is what its context
described. No prompt change would have rescued that answer: the
sentence holding it never left the database. By the end of this
lesson you will know the four stages where a passage gets lost,
what to do at each one, and how to measure retrieval on its own so
you find the leak before your users do.
Retrieval here means the whole pipeline that turns a question into the handful of passages you paste into the prompt, and it has four stages that can each drop the answer.
Chunking
Documents are cut into pieces. An answer split across two pieces is damaged before you index a thing.
Indexing
Each piece becomes something searchable — an embedding (a list of numbers positioned so that similar meanings sit close together), a keyword index, or both.
What the index cannot represent, you cannot later search for.
Searching
Chunks are scored against the question. A scoring function subtly wrong about "relevant" puts the answer at rank 40 instead of rank 2.
Cutting
Only the top few survive, because context is a budget rather than a bucket. Rank 40 does not survive a cut at 5.
The funnel only ever narrows. It is a relay race in which every runner can drop the baton and none can pick one up: nothing downstream recovers a passage an earlier stage lost, not a stronger model, not a longer prompt, not a sterner instruction to use the provided documents. That is why the last section of this lesson — measuring recall — pays for all the others.
The default first build splits documents every thousand characters. It is fast, it is three lines, and it fails in two ways that look nothing like each other from outside.
The first is the severed procedure. Your reset instructions run to seven steps; the boundary lands after step five. The chunk winning the search for "how do I reset the controller" is a fluent, confident passage that stops at the amber status light and never mentions the two steps finishing the job. The model cannot tell it holds half a procedure, so it presents half a procedure as the answer.
The second is the orphaned passage. Here is a chunk exactly as the index sees it:
Set this to 30 seconds. Values above 120 are rejected by the
firmware, and a value of 0 disables the check entirely.Set what to 30 seconds, on which product? The heading two levels up said "RX-2200 > Network > Heartbeat interval", and it is not in the chunk. Embedded alone, this passage is about seconds and rejected values, so it will never be the top hit for "what should the RX-2200 heartbeat be" — the words the searcher used are not in it.
Bad — splits on character count, so boundaries land mid-procedure and every chunk arrives at the index stripped of the heading that gave it a subject.
def chunk(document_text: str, size: int = 1000) -> list[str]:
return [
document_text[start : start + size]
for start in range(0, len(document_text), size)
]Good — splits on the document's own structure, and prefixes each piece with the heading path it came from.
def chunk_section(section: Section) -> list[Chunk]:
breadcrumb = " > ".join(section.heading_path)
chunks: list[Chunk] = []
current = ""
for paragraph in section.body.split("\n\n"):
if current and len(current) + len(paragraph) > 1200:
chunks.append(Chunk(breadcrumb, current, section.id))
current = ""
current += paragraph + "\n\n"
if current:
chunks.append(Chunk(breadcrumb, current, section.id))
return chunksThe single difference is that the second splitter knows the document has a shape. Both failures above come from the first one not knowing: a numbered list is invisible to a character counter, and a heading is just more characters. The cost is a system that returns text genuinely about the right topic and still cannot answer the question — the hardest kind of bug to see, because every chunk you inspect looks fine.
Two notes on the good version. A heading breadcrumb costs forty characters per chunk and buys back every product and feature name the author wrote only once, in a heading. And treat tables, code blocks and numbered lists as atomic: split them and you get rows without a header row, which is worse than not retrieving them.
An embedding places a passage in space so that things meaning similar things end up near each other. That is exactly what you want for "how do I stop the fan spinning up at night" matching a passage titled "Thermal throttling schedules", and it is why vector search feels like magic in a demo. It is also why the demo hid the problem: the same smoothing over surface form makes an embedding a poor instrument for exact strings.
E-412 and E-413 are different failures with
different fixes, and land in nearly the same place in the space.retry_backoff_ms.None of these were common enough in training to earn a meaningful
position. The model producing the embedding sees E-412 as a
scrap of technical-looking text about errors, which describes
every other error passage in your corpus equally well.
Finds the topic.
Matches "how do I stop the fan spinning up at night" to a passage titled "Thermal throttling schedules".
And puts E-412 and E-413 in nearly the same place, which
is exactly wrong for two different failures.
Finds the string.
BM25, an inverted index, Postgres tsvector — whatever your
database already offers.
It has no idea that "fan noise" and "thermal throttling" are
related, and it will never miss the one document containing
the literal E-412.
Hybrid retrieval means issuing the query to both indexes and merging the two result lists into one. The merge is where people go wrong, so be deliberate about it.
The tempting move is to add the scores together, perhaps with a weight. Do not. A cosine similarity sits around 0 to 1 and clusters tightly; a BM25 score is unbounded and depends on how rare the query terms are in your corpus. They are not on the same scale, they are not on the same kind of scale, and the relationship between them shifts from query to query. A weight you tune on ten questions will be wrong on the eleventh.
Fuse the ranks instead. Reciprocal rank fusion discards the scores and keeps only each result's position in its own list:
from collections import defaultdict
def reciprocal_rank_fusion(rankings, k: int = 60):
scores: dict[str, float] = defaultdict(float)
for ranking in rankings:
for position, chunk_id in enumerate(ranking, start=1):
scores[chunk_id] += 1 / (k + position)
return sorted(scores, key=scores.get, reverse=True)
candidates = reciprocal_rank_fusion([
[hit.chunk_id for hit in vector_search(embed(question), 50)],
[hit.chunk_id for hit in keyword_search(question, 50)],
])The constant k damps the influence of the very top positions, so
a chunk ranked 2 in both lists beats one ranked 1 in a single list
and absent from the other. Sixty is the conventional default and
is rarely worth tuning. The property is what matters: a passage
both methods find plausible outranks a passage one method loves.
The E-412 chunk now arrives through the keyword lane even when
the embedding cannot see it.
Here is a limitation of every embedding index, structural rather than a quality problem. Chunk vectors are computed at index time, long before anyone asks a question, so the comparison is between two summaries made independently — your question squashed into one vector, the passage into another — and the scoring never considers them together. That design is what makes searching millions of chunks fast.
A reranker (a cross-encoder) makes the opposite trade. It reads the question and one passage at the same time and scores how well that passage answers that question. It is far more accurate and far too slow to run over a whole corpus. So you use each for what it is good at: cheap search casts a wide net, the reranker chooses within it.
candidate_ids = candidates[:50] # from the fusion above
passages = load_chunks(candidate_ids)
scored = rerank(query=question, passages=passages)
context = [passage for passage, _score in scored[:8]]The shift in habit is the number 50. A first build asks the vector index for the top 5 and hands all 5 to the model, which makes the embedding's opinion final. Ask for 50 and the embedding only has to get the answer somewhere in the room; the reranker, far better at judging, decides the order. Recall is the first stage's job and precision is the second's, and trying to get both from one stage is how you get neither. The bill is a second model call and its latency per query, and it tends to buy the largest single quality jump available in a retrieval pipeline.
Alongside the text, store what you know about each chunk: source document, product version, language, publication date, and which audience is allowed to see it. Metadata filters turn those fields into hard constraints, and the place you apply them decides whether they work.
Bad — filters the eight results the search already chose, so a query whose top eight are all old documentation returns nothing at all.
hits = vector_search(embed(question), limit=8)
hits = [hit for hit in hits if hit.product_version == "4.x"]Good — hands the constraint to the index, which ranks only chunks that already satisfy it.
hits = vector_search(
embed(question),
limit=8,
where={"product_version": "4.x"},
)Post-filtering looks harmless in testing, where most queries leave a couple of survivors, and collapses in exactly the case you care about most: an old, heavily documented feature where 3.x text dominates the rankings. You retrieve eight passages, filter to zero, send an empty context, and the model answers from whatever it happens to remember. The bad version never fails loudly — it quietly turns your retrieval feature back into an ordinary chatbot for a set of questions you will never notice.
Two filters earn their keep immediately. Version or date, because a superseded procedure that is textually perfect is worse than no result — it is wrong with confidence. And permissions, because once a corpus mixes audiences, which chunks a user may see is a correctness requirement, not a ranking preference.
This is the section people skip, and skipping it is why teams spend a fortnight rewriting prompts to fix a chunking bug.
When an answer is wrong you are looking at two systems chained together: retrieval fetched some passages, then the model wrote from them. Judge only the final text and you cannot tell which one failed — and the two have opposite fixes. So measure the first one alone. You need a small set of real questions paired with the chunks that actually answer them.
For each question, find the chunk ids that genuinely contain the answer and write them down. That list is your gold set, and building it is a couple of afternoons of unglamorous reading that pays for itself in the first week.
Recall@k then asks one question: within the top k results, did an answer-bearing chunk come back at all?
def recall_at_k(examples, k: int) -> float:
found = 0
for question, gold_chunk_ids in examples:
hits = retrieve(question, limit=k)
retrieved = {hit.chunk_id for hit in hits}
if retrieved & set(gold_chunk_ids):
found += 1
return found / len(examples)Note what it does not measure: nothing about wording, nothing about tone, nothing needing a human or a second model to judge. It is a set-membership test — deterministic, and it runs in seconds.
Bad — one assertion covering both systems, so a red test could mean the chunk was missing or that the model phrased it differently.
def test_error_code_lookup():
answer = ask("what does E-412 mean")
assert "coolant" in answer.lower()Good — asserts on what retrieval returned, so a failure has exactly one meaning.
def test_error_code_lookup():
hits = retrieve("what does E-412 mean", limit=8)
assert "handbook/errors#e-412" in {h.chunk_id for h in hits}The difference is which system is under test. When the first goes red you have no idea where to start, so you start where it is easy to start — the prompt — and burn days there while the passage sits unretrieved in the database. When the second goes red, the chunk was not fetched, and you know to look at chunking, at the index, or at hybrid search.
Now measure at two points in the funnel, and the number becomes a diagnosis rather than a grade:
recall@50 recall@8 the stage that is losing the answer
--------- -------- -----------------------------------
low low chunking, or you need keyword search
high low the reranker or the fusion is wrong
high high retrieval is healthy — look downstreamA low recall@50 means the answer never entered the candidate set and no amount of reranking invents it: go back to chunk boundaries, heading breadcrumbs, and whether the exact terms in those failing questions are reachable by keyword at all. Two high numbers mean the right material is in the context window and whatever is wrong is happening after it.
Track both on every change to the pipeline — fusing ranks, adding a reranker, resizing chunks — so you can see which way each one moved them. Turning this into a CI suite with thresholds, and judging the generated answer too, is the subject of "Evaluating an AI Feature"; here, recall is enough.
Split on structure, never character count # headings, steps, tables
Keep tables and numbered lists whole # half a table is noise
Prefix each chunk with its heading path # "RX-2200 > Network > ..."
Store id, source, version, date, audience # you will filter on these
Vector search finds meaning # paraphrase, synonym, topic
Keyword search finds exact strings # E-412, SKUs, config keys
Run both — their weaknesses are opposite # this is hybrid retrieval
Fuse by rank, never by raw score # RRF, k=60, no tuning
Fetch 30-50 candidates, not 5 # recall is this stage's job
Rerank with a cross-encoder, keep ~8 # precision is that one's job
Filter inside the query, never after # post-filtering empties it
Re-embed the whole corpus on model change # mixed vectors fail silently
Gold set: 30-50 real questions + chunk ids # from logs, not imagination
recall@k = did any gold chunk come back # deterministic, seconds
Measure recall at candidates AND after cut # the pair is the diagnosis
low / low -> chunking or missing keyword search
high / low -> reranking or fusion
high / high -> retrieval is fine; the fault is downstreamYou can now build a retrieval pipeline that actually retrieves: chunks that describe themselves, two indexes whose blind spots do not overlap, a generous candidate set narrowed by a reranker, filters enforced where they cannot be undone, and a recall number naming the stage to fix. The right material reaches the context window.
Getting it there is half the job. The other half is what the model does with it — whether the answer traces back to the passage it came from, and whether the model says "the documents do not cover this" instead of filling the gap from memory. That is "Grounding and Citation", the next lesson, and it rests entirely on the work you just did: you cannot cite a source you never fetched.
Before you move on, do this. Take the ten questions your product has been worst at, find the chunk that should have answered each, and run recall@8 against today's pipeline. Whatever fraction comes back is the ceiling on your feature's quality, and it is almost always lower than the team guessed.