intermediate30 minby DevFox

Retrieval That Actually Retrieves

Why naive vector search disappoints: chunking that destroys meaning, embeddings that miss exact terms, and no reranking. Hybrid retrieval, chunk design, and measuring recall before you blame the model.

Your first retrieval demo went beautifully. You split the support handbook into chunks, embedded them, asked "how do I reset the controller", and the right paragraph came back at rank one. You shipped it that afternoon.

Then a support engineer typed "what does E-412 mean" and got three passages about error handling in general, not one of which contained the string E-412. The model wrote a confident paragraph about cable seating, because that is what its context described. No prompt change would have rescued that answer: the sentence holding it never left the database. By the end of this lesson you will know the four stages where a passage gets lost, what to do at each one, and how to measure retrieval on its own so you find the leak before your users do.

Retrieval is a funnel, and it only leaks

Retrieval here means the whole pipeline that turns a question into the handful of passages you paste into the prompt, and it has four stages that can each drop the answer.

Chunking
Documents are cut into pieces. An answer split across two pieces is damaged before you index a thing.
Indexing
Each piece becomes something searchable — an embedding (a list of numbers positioned so that similar meanings sit close together), a keyword index, or both.
What the index cannot represent, you cannot later search for.
Searching
Chunks are scored against the question. A scoring function subtly wrong about "relevant" puts the answer at rank 40 instead of rank 2.
Cutting
Only the top few survive, because context is a budget rather than a bucket. Rank 40 does not survive a cut at 5.

Four stages, four ways to lose the passage that had the answer in it

The funnel only ever narrows. It is a relay race in which every runner can drop the baton and none can pick one up: nothing downstream recovers a passage an earlier stage lost, not a stronger model, not a longer prompt, not a sterner instruction to use the provided documents. That is why the last section of this lesson — measuring recall — pays for all the others.

Chunking is where meaning gets cut

The default first build splits documents every thousand characters. It is fast, it is three lines, and it fails in two ways that look nothing like each other from outside.

The first is the severed procedure. Your reset instructions run to seven steps; the boundary lands after step five. The chunk winning the search for "how do I reset the controller" is a fluent, confident passage that stops at the amber status light and never mentions the two steps finishing the job. The model cannot tell it holds half a procedure, so it presents half a procedure as the answer.

The second is the orphaned passage. Here is a chunk exactly as the index sees it:

Set what to 30 seconds, on which product? The heading two levels up said "RX-2200 > Network > Heartbeat interval", and it is not in the chunk. Embedded alone, this passage is about seconds and rejected values, so it will never be the top hit for "what should the RX-2200 heartbeat be" — the words the searcher used are not in it.

Bad — splits on character count, so boundaries land mid-procedure and every chunk arrives at the index stripped of the heading that gave it a subject.

Good — splits on the document's own structure, and prefixes each piece with the heading path it came from.

The single difference is that the second splitter knows the document has a shape. Both failures above come from the first one not knowing: a numbered list is invisible to a character counter, and a heading is just more characters. The cost is a system that returns text genuinely about the right topic and still cannot answer the question — the hardest kind of bug to see, because every chunk you inspect looks fine.

Two notes on the good version. A heading breadcrumb costs forty characters per chunk and buys back every product and feature name the author wrote only once, in a heading. And treat tables, code blocks and numbered lists as atomic: split them and you get rows without a header row, which is worse than not retrieving them.

Embeddings know the topic, not the token

An embedding places a passage in space so that things meaning similar things end up near each other. That is exactly what you want for "how do I stop the fan spinning up at night" matching a passage titled "Thermal throttling schedules", and it is why vector search feels like magic in a demo. It is also why the demo hid the problem: the same smoothing over surface form makes an embedding a poor instrument for exact strings.

Error codes: E-412 and E-413 are different failures with different fixes, and land in nearly the same place in the space.
Identifiers: part numbers, SKUs, order references, commit hashes, config keys like retry_backoff_ms.
Proper nouns: a customer's surname, an internal service name, a version number that changes one digit at a time.

None of these were common enough in training to earn a meaningful position. The model producing the embedding sees E-412 as a scrap of technical-looking text about errors, which describes every other error passage in your corpus equally well.

Vector search

Finds the topic.

Matches "how do I stop the fan spinning up at night" to a passage titled "Thermal throttling schedules".

And puts E-412 and E-413 in nearly the same place, which is exactly wrong for two different failures.

Keyword search

Finds the string.

BM25, an inverted index, Postgres tsvector — whatever your database already offers.

It has no idea that "fan noise" and "thermal throttling" are related, and it will never miss the one document containing the literal E-412.

Each method's weakness is the other's strength, which is the argument for running both rather than choosing

Hybrid search: run both, fuse the ranks

Hybrid retrieval means issuing the query to both indexes and merging the two result lists into one. The merge is where people go wrong, so be deliberate about it.

The tempting move is to add the scores together, perhaps with a weight. Do not. A cosine similarity sits around 0 to 1 and clusters tightly; a BM25 score is unbounded and depends on how rare the query terms are in your corpus. They are not on the same scale, they are not on the same kind of scale, and the relationship between them shifts from query to query. A weight you tune on ten questions will be wrong on the eleventh.

Fuse the ranks instead. Reciprocal rank fusion discards the scores and keeps only each result's position in its own list:

The constant k damps the influence of the very top positions, so a chunk ranked 2 in both lists beats one ranked 1 in a single list and absent from the other. Sixty is the conventional default and is rarely worth tuning. The property is what matters: a passage both methods find plausible outranks a passage one method loves. The E-412 chunk now arrives through the keyword lane even when the embedding cannot see it.

Retrieve generously, then rerank

Here is a limitation of every embedding index, structural rather than a quality problem. Chunk vectors are computed at index time, long before anyone asks a question, so the comparison is between two summaries made independently — your question squashed into one vector, the passage into another — and the scoring never considers them together. That design is what makes searching millions of chunks fast.

A reranker (a cross-encoder) makes the opposite trade. It reads the question and one passage at the same time and scores how well that passage answers that question. It is far more accurate and far too slow to run over a whole corpus. So you use each for what it is good at: cheap search casts a wide net, the reranker chooses within it.

The shift in habit is the number 50. A first build asks the vector index for the top 5 and hands all 5 to the model, which makes the embedding's opinion final. Ask for 50 and the embedding only has to get the answer somewhere in the room; the reranker, far better at judging, decides the order. Recall is the first stage's job and precision is the second's, and trying to get both from one stage is how you get neither. The bill is a second model call and its latency per query, and it tends to buy the largest single quality jump available in a retrieval pipeline.

Filters belong inside the query

Alongside the text, store what you know about each chunk: source document, product version, language, publication date, and which audience is allowed to see it. Metadata filters turn those fields into hard constraints, and the place you apply them decides whether they work.

Bad — filters the eight results the search already chose, so a query whose top eight are all old documentation returns nothing at all.

Good — hands the constraint to the index, which ranks only chunks that already satisfy it.

Post-filtering looks harmless in testing, where most queries leave a couple of survivors, and collapses in exactly the case you care about most: an old, heavily documented feature where 3.x text dominates the rankings. You retrieve eight passages, filter to zero, send an empty context, and the model answers from whatever it happens to remember. The bad version never fails loudly — it quietly turns your retrieval feature back into an ordinary chatbot for a set of questions you will never notice.

Two filters earn their keep immediately. Version or date, because a superseded procedure that is textually perfect is worse than no result — it is wrong with confidence. And permissions, because once a corpus mixes audiences, which chunks a user may see is a correctness requirement, not a ranking preference.

Measure retrieval before you blame the model

This is the section people skip, and skipping it is why teams spend a fortnight rewriting prompts to fix a chunking bug.

When an answer is wrong you are looking at two systems chained together: retrieval fetched some passages, then the model wrote from them. Judge only the final text and you cannot tell which one failed — and the two have opposite fixes. So measure the first one alone. You need a small set of real questions paired with the chunks that actually answer them.

For each question, find the chunk ids that genuinely contain the answer and write them down. That list is your gold set, and building it is a couple of afternoons of unglamorous reading that pays for itself in the first week.

Recall@k then asks one question: within the top k results, did an answer-bearing chunk come back at all?

Note what it does not measure: nothing about wording, nothing about tone, nothing needing a human or a second model to judge. It is a set-membership test — deterministic, and it runs in seconds.

Bad — one assertion covering both systems, so a red test could mean the chunk was missing or that the model phrased it differently.

Good — asserts on what retrieval returned, so a failure has exactly one meaning.

The difference is which system is under test. When the first goes red you have no idea where to start, so you start where it is easy to start — the prompt — and burn days there while the passage sits unretrieved in the database. When the second goes red, the chunk was not fetched, and you know to look at chunking, at the index, or at hybrid search.

Now measure at two points in the funnel, and the number becomes a diagnosis rather than a grade:

A low recall@50 means the answer never entered the candidate set and no amount of reranking invents it: go back to chunk boundaries, heading breadcrumbs, and whether the exact terms in those failing questions are reachable by keyword at all. Two high numbers mean the right material is in the context window and whatever is wrong is happening after it.

Track both on every change to the pipeline — fusing ranks, adding a reranker, resizing chunks — so you can see which way each one moved them. Turning this into a CI suite with thresholds, and judging the generated answer too, is the subject of "Evaluating an AI Feature"; here, recall is enough.

A cheat sheet to keep

Where to go next

You can now build a retrieval pipeline that actually retrieves: chunks that describe themselves, two indexes whose blind spots do not overlap, a generous candidate set narrowed by a reranker, filters enforced where they cannot be undone, and a recall number naming the stage to fix. The right material reaches the context window.

Getting it there is half the job. The other half is what the model does with it — whether the answer traces back to the passage it came from, and whether the model says "the documents do not cover this" instead of filling the gap from memory. That is "Grounding and Citation", the next lesson, and it rests entirely on the work you just did: you cannot cite a source you never fetched.

Before you move on, do this. Take the ten questions your product has been worst at, find the chunk that should have answered each, and run recall@8 against today's pipeline. Whatever fraction comes back is the ceiling on your feature's quality, and it is almost always lower than the team guessed.