What retrieval is for
Retrieval-augmented generation is a plain idea: find the relevant material, put it in the prompt, and ask the model to answer from it. The reason it matters is that it fixes several different problems at once, and the reason it disappoints is that teams expect it to fix a few it cannot.
It supplies facts the model does not have. Your contracts, your policies, your tickets, your product documentation. None of it was in the training corpus and no amount of prompting will produce it.
It makes currency a solved problem. A model's knowledge ends at its training cut-off. An index can be updated this morning, which turns a model property into an operational one.
It gives you provenance. The system can show which document a statement came from. That changes the product from something a user has to trust into something a user can check, and it is the single most valuable property retrieval adds.
It gives you a place to put access control. Which documents a particular user may see is a question with an existing answer in most organisations. Enforcing it at the retrieval step is tractable; trying to enforce it inside a model is not.
And it is usually cheaper than the alternative. Supplying four relevant passages costs a fraction of supplying an entire document set, and as the previous course showed, it is also more reliable than a very long context.
What it does not fix. It does not make the model reason better. It does not stop the model embellishing beyond the supplied text. It does not impose a format. It does not help with questions whose answer is a calculation over structured data, where a query against a database is the right tool.
The published origin is worth knowing. The method was introduced as combining a parametric memory - the model's weights - with a non-parametric one, an index that can be swapped or updated without retraining. That framing is still the clearest way to think about it: you are adding a memory you control to a model you do not.
Most failures are retrieval failures. When a system of this kind gives a bad answer, the usual cause is that the right passage was not in the prompt. Teams spend their effort on the generation step because that is the visible part, and the fault is almost always upstream.
So the structure of this course follows the pipeline. Ingestion, chunking, embeddings, search, hybrid ranking, re-ranking, prompt assembly, query handling, access control, evaluation and operation. The engineering is in the first half and the credibility is in the second.
The pipeline, and where it breaks
It is worth laying out the whole chain once, with the characteristic failure of each stage, because diagnosing a bad answer means knowing which link to inspect.
Ingestion. Documents are collected and converted to text. Fails by producing garbled text from a difficult file, losing table structure, including page furniture, or silently missing a document set nobody checked.
Chunking. Text is split into passages. Fails by cutting a table or a clause in half, by producing chunks too large to be precise or too small to be self-contained, or by dropping the heading that said what the passage was about.
Embedding. Each chunk becomes a vector. Fails when the embedding model does not suit the domain, when the text is in a language it handles poorly, or when the query and the document are embedded in incompatible ways.
Indexing. Vectors are stored in a structure that supports fast nearest-neighbour search. Fails through approximation set too aggressively, stale entries after updates, or a filter that cannot be applied efficiently.
Retrieval. The query is embedded and the nearest chunks returned. Fails by returning passages about the right topic that do not contain the answer, by missing an exact identifier that keyword search would have found, or by returning several near-duplicates instead of several different facts.
Re-ranking. A more expensive model reorders the candidates. Fails by being skipped, which is the most common configuration in systems that do not work well.
Assembly. The chosen passages, instruction and question are composed into a prompt. Fails through a token budget that silently truncates, an order that buries the evidence, or a missing instruction about what to do when the answer is absent.
Generation. The model answers. Fails by going beyond the passages, by merging two documents into one claim, or by answering confidently from its own training when the passages do not say.
Citation and presentation. The answer is shown with its sources. Fails by citing a document that does not support the sentence, which is worse than no citation because it manufactures trust.
The diagnostic discipline that follows. When an answer is wrong, first ask whether the right passage was retrieved at all. If it was not, everything downstream is irrelevant. If it was, the fault is in assembly or generation. Teams that do not separate these two questions tune the wrong half of the system for weeks.
Ingestion, the part nobody budgets for
The least interesting stage determines the ceiling on everything above it. A system cannot retrieve what ingestion mangled, and most disappointing deployments are explained here.
Document formats are hostile. Word processor files carry structure that converts unevenly. Spreadsheets have meaning in their layout. Slides are fragments. Scanned documents are images. Each needs a different path and a decision about what to do when extraction fails.
Portable documents are the usual culprit. Text in such files has position but not necessarily order. Two-column layouts interleave. Headers and footers repeat on every page. Tables become a stream of numbers with no relationship. A naive extraction of a hundred-page agreement often produces text no human could read, and the index then contains that.
Tables need a deliberate decision. Flattening a table into prose loses the row and column relationship that carried the meaning. Keeping it as structured text, converting each row into a sentence, or extracting it into a separate queryable store are all defensible; doing nothing is not.
Scanned material needs recognition, and it has an error rate. Numbers, names and identifiers are where recognition fails most and where accuracy matters most. If a corpus is substantially scanned, measure the recognition quality before building anything on it.
Strip the furniture. Repeated headers, footers, page numbers, confidentiality notices and navigation text add noise to every chunk and bias similarity towards whichever document has the most boilerplate.
De-duplicate. Document sets accumulate versions: a draft, a final, a signed copy, a scan of the signed copy. An index containing all four returns four copies of the same passage and crowds out the other facts. Near-duplicate detection at ingestion is cheap and prevents a class of user-visible failure.
Capture metadata while you have it. Source, title, document type, author, date, version, access group, section heading. Every one of these is useful later for filtering, for ranking, for citation and for access control, and none can be recovered once the text is in the index.
Decide what wins when versions conflict. If the index holds last year's policy and this year's, a query will find both, and the model has no way to prefer the current one. Either keep only the current version or make the date available and instruct the system to prefer it.
And log what you ingested. Counts by source, extraction failures, documents skipped, pages recognised. The most common silent fault in these systems is a document set that never made it in, and the only way to notice is a number somebody looks at.
Chunking
Splitting documents into passages is where the most consequential cheap decisions get made. There is no universally right chunk size, and there are several clearly wrong ways to choose one.
Why chunk at all. Embeddings represent a passage in one vector, and a long passage averages away the specific. Retrieval also has to return something that fits in the prompt alongside everything else. Chunking is a compromise between representational precision and self-containedness.
Too large and precision suffers. A chunk covering five topics has a vector that is near none of them. It will be retrieved for loosely related queries and will consume budget delivering four irrelevant topics with the one that mattered.
Too small and context is lost. A two-sentence chunk may be perfectly on topic and incomprehensible without its heading or the clause before it. The model then answers from a fragment, which is a common cause of confidently wrong output.
Split on structure, not on character counts. Headings, sections, articles, clauses, list items and paragraph boundaries are meaningful; a fixed window is not. A splitter that respects the document's own structure outperforms a fixed one on almost every corpus.
Keep the heading with the chunk. Prepending the document title and the section path to each passage costs a few tokens and improves both retrieval and the model's ability to interpret the passage. It is the highest-return change available at this stage.
Overlap, modestly. A small overlap between adjacent chunks prevents an answer that straddles a boundary from being lost. Large overlaps inflate the index and return near-duplicates.
The parent-child pattern is worth knowing. Index small, precise chunks for matching, but supply the larger parent section to the model once a child matches. You get the precision of small chunks and the context of large ones, at the cost of a slightly more complex retrieval step.
Some content should not be chunked by text at all. Code, tables, question-and-answer pairs and structured records have natural units. Use them.
Measure chunking rather than debating it. Build a set of real questions with the passage that answers each, then measure how often retrieval returns it at different chunk sizes and strategies. This is a half-day experiment that settles an argument teams otherwise have for months.
Embeddings for retrieval
The embeddings used for retrieval are not the ones inside a generative model. They are produced by a model trained specifically to put related text near related text, and the distinction matters.
A bi-encoder embeds separately and compares cheaply. The query and the document are each turned into one vector, independently, and similarity is a dot product or a cosine. Independence is the point: document vectors can be computed once, in advance, for millions of passages.
Published dense retrieval established that this beats keyword matching on natural-language questions. Training on question-passage pairs produced a model whose vectors put a question near the passage that answers it, which is not the same as putting it near passages that share its words.
Sentence-level training was the enabling step. Work on siamese training of a bidirectional encoder produced sentence vectors usable for direct comparison, which earlier models did not provide. That is why an embedding model is a different artefact from a language model.
Queries and documents are asymmetric. A short question and a long passage are different kinds of text. Many embedding models are trained with distinct instructions or prefixes for each side, and using the wrong one quietly degrades retrieval. Read the model's own instructions; this is a frequent and invisible mistake.
Domain fit matters more than benchmark rank. A model that leads a general leaderboard may be mediocre on medical text, legal drafting, or a corpus full of product codes. Evaluate on your own questions; the published heterogeneous benchmark exists precisely because performance does not transfer reliably between domains.
Dimension is a cost decision. More dimensions capture more distinctions and cost more memory and comparison time. Many current models support producing shorter vectors with modest loss, which is worth measuring if your index is large.
Multilingual retrieval needs a multilingual model. Embedding a Swahili question against an English corpus works only if the model was trained to place translations near each other. If you serve several languages, test each one separately rather than assuming.
Normalise and be consistent. Whatever is done at indexing must be done identically at query time: the same model, the same version, the same prefixes, the same normalisation. A silent model upgrade invalidates an index, and the symptom is a system that gets quietly worse.
And treat the vectors as personal data where the text was. Embeddings retain a great deal of their source, and text can be substantially reconstructed from them. A vector store holding customer correspondence is holding customer correspondence, with the same obligations.
Searching the vectors
Finding the nearest vectors to a query among millions is a solved problem with well-understood trade-offs. Knowing them prevents two common faults: paying for exactness you do not need, and losing recall you did not know you had lost.
Exact search is linear and sometimes correct. Comparing the query against every vector is simple and guarantees the true nearest neighbours. For tens of thousands of chunks it is fast enough, and the complexity of an approximate index is not worth paying.
Approximate search trades recall for speed. The published approaches fall into two families. Graph-based methods build a navigable structure in which search walks towards the query, with the hierarchical small-world construction the most widely used. Partition-based methods cluster the vectors and search only the nearest clusters, often combined with compression of each vector.
Compression is where the memory saving is. Quantising vectors to fewer bits per dimension reduces an index dramatically, with a measurable accuracy cost. Published work on billion-scale search on accelerators is the standard reference for how far this can be pushed.
Every approximate index has a knob you must set and measure. How many neighbours to explore, how many clusters to probe. Turning it down makes search faster and quietly loses results. The only way to know your operating point is to measure recall against exact search on a sample.
Filtering interacts badly with approximation. Restricting results to a document type, a date range or a permission group is easy to express and hard to execute efficiently. Filtering after the search can return nothing; filtering during it requires index support. This is the question to ask any vector database before choosing it, and lesson eleven explains why it is a security matter and not only a performance one.
Updates and deletions are not free. Graph indexes do not always handle deletion gracefully, and some require periodic rebuilding. If your corpus changes daily, that is a requirement to state at the start rather than discover later.
Index size is predictable arithmetic. Number of chunks times dimensions times bytes per value, plus the index overhead. Do that multiplication before choosing a chunk size, because a small chunk size multiplies everything.
Hosted and embedded options both exist. A library in your process is simpler and limited by one machine. A service adds operational cost and gives you filtering, replication and persistence. Choose on corpus size, change rate and whether you need permission filtering at query time.
And keep exact search available for evaluation. Comparing your production index against exact results on a sample of queries is the only way to separate a retrieval model problem from an index configuration problem.
Keyword search did not become obsolete
The most common unforced error in these systems is replacing a working keyword index with vectors alone. The two methods fail in different places, which is exactly why they belong together.
Keyword ranking is a mature technology. The standard probabilistic ranking function scores a document on term frequency, inverse document frequency and length normalisation. It is fast, requires no model, needs no training on your corpus, and is strong at exactly the queries vectors handle worst.
Exact strings are where vectors fail. A part number, a case citation, an account reference, a function name, a defined term, a person's surname. An embedding places these approximately; a keyword index finds them exactly. A user searching for an identifier expects the document containing it, not a document about similar things.
Rare terms are the same problem. Inverse document frequency gives weight to a word that appears in three documents out of a million. A dense vector has no comparable mechanism, and a distinctive term can be averaged into insignificance.
Vectors win on paraphrase and intent. A question phrased in ordinary language, with none of the corpus's vocabulary, is where dense retrieval earns its place. The two methods are complements, not competitors.
So retrieve with both and combine. Run the query against the keyword index and the vector index, then merge. Reciprocal rank fusion - scoring each result by the reciprocal of its rank in each list and summing - is the usual method, needs no tuning, and is robust because it uses ranks rather than incomparable scores.
Hybrid retrieval is close to a free improvement. On most corpora it beats either method alone, at the cost of running one more query and a few lines of merging. Teams who skip it and then struggle with recall are the single most common pattern in this subject.
Metadata filters belong in the same step. Document type, date, author, access group, status. A filter that removes two thirds of the corpus before ranking improves both precision and latency, and it is how a question about this year's policy avoids last year's.
Keep the keyword index honest. Stemming, stop words, synonyms for your domain's vocabulary, and the identifiers tokenised so they survive. A keyword index that splits a part number into fragments provides none of the benefit above.
And remember what retrieval is being asked to do. The goal is not the best-ranked list; it is getting the passage that contains the answer into the top handful, every time. Hybrid retrieval raises that floor, which is what matters to a user who only asks once.
Re-ranking
The retrieval stage is optimised to be fast over millions of passages, which means it is approximate about relevance. A second, slower stage over a handful of candidates fixes most of what the first stage gets wrong, and it is the most frequently omitted component in these systems.
The two-stage pattern. Retrieve perhaps fifty candidates cheaply, then score each one properly and keep the best four or five. The first stage optimises recall, the second precision, and each is allowed to be bad at the other's job.
A cross-encoder is what does the scoring. Instead of embedding query and passage separately, it processes them together and outputs a relevance score. Because the two texts attend to each other, it can tell that a passage mentions the right entity in the wrong role, which a dot product between two independent vectors cannot.
It is accurate and it does not scale. Every candidate needs a forward pass, so it is usable over fifty passages and impossible over five million. That asymmetry is the entire reason for two stages. Published work established this pattern for passage ranking and it has not been displaced.
The quality gain is usually larger than any prompt change. On most corpora, adding re-ranking moves more questions from wrong to right than any amount of instruction tuning in the prompt, because it changes what evidence the model gets rather than how it is asked.
Choose the candidate count deliberately. Too few and re-ranking cannot rescue a passage the first stage buried. Too many and latency and cost rise without benefit. Measure recall at the candidate count to find where the right passage is actually present, then re-rank that many.
A model can re-rank without a dedicated ranker. Asking a small generative model to score each passage's relevance works and costs more. Dedicated cross-encoders are cheaper and faster for the same quality, and this is one of the few places where the specialised tool is clearly better.
Diversity deserves a thought. Five passages saying the same thing are worse than three distinct ones, because budget is spent on repetition. A de-duplication step on the re-ranked list, or a selection method that penalises similarity to what is already chosen, recovers that budget.
Keep the scores. The re-ranker's score is a usable confidence signal: if the best passage scores poorly, the corpus probably does not contain the answer, and that is the moment to say so rather than to generate. Systems that expose this threshold have a principled no-answer path, which is the subject of the next lesson.
Assembling the prompt
Retrieval has produced a handful of passages. What happens next determines whether the user gets a grounded answer or a plausible one, and almost all of it is ordinary engineering.
Put the instruction first and keep it stable. It reads better, it caches better, and it sits where the model attends most reliably. The passages follow, and the user's question goes last, immediately before the answer.
Label every passage. A document title, a section, a date and an identifier before each extract. This gives the model something to cite, lets it prefer the current version over an old one, and makes the assembled prompt readable by a human debugging it.
Order with the middle effect in mind. The published finding that material in the centre of a long input is used less reliably applies directly here. Place the highest-ranked passage first or last rather than in the middle of the set, and keep the set small enough that the question does not arise.
Give an explicit instruction about absence. Tell the model to answer only from the passages and to state plainly when they do not contain the answer. This single sentence prevents a large share of fabrication, and it works better when paired with a retrieval score threshold that stops the call entirely.
Require citation at the sentence level. Asking for a reference after each claim produces answers that can be checked. Then verify the citations mechanically: does the cited passage exist in the prompt, and does it contain the claim's key terms. A system that cites a passage it was not given has a bug worth catching automatically.
Budget the window explicitly. Instruction, passages, conversation history and the answer come out of one allowance. Decide in advance how many tokens each component may use, enforce it in code, and drop passages rather than truncating mid-passage.
Treat passage text as data, not instruction. A document in the corpus may contain text that reads as an instruction, placed there deliberately or by accident. Delimit the passages clearly, instruct the model that their content is reference material only, and do not let a retrieved document change what the system does. The course on securing systems against agents covers the general case.
Keep the conversation out of the retrieval query. The question sent to the index should be a resolved, standalone query, which is the next lesson. The conversation belongs in the prompt, not in the embedding.
And log the assembled prompt. With token counts by component and the identifiers of the chosen passages. When an answer is wrong, this log tells you in one glance whether the evidence was there, and nothing else does.
Handling the question
The query that arrives is rarely the query that should be sent to the index. The transformations in this lesson are cheap and account for a large share of the difference between a demonstration and a working system.
Resolve the conversation first. A follow-up such as what about the second one is meaningless to an index. Rewriting it into a standalone question, using the conversation so far, is a small call to a model and it is essential in any multi-turn system.
Decompose questions that contain several. A question asking how two policies differ needs passages about both, and one embedding of the whole question may retrieve neither well. Splitting it, retrieving for each part and merging is substantially better.
Expand vocabulary where the corpus has its own. A user asking about sick leave may need documents that say absence management. A short expansion step, or a maintained synonym list for your domain, closes a gap that embeddings only partly bridge.
Extract filters from the question. Dates, document types, parties, jurisdictions and statuses mentioned in a question should become metadata filters rather than being left to similarity. A question about last quarter's figures should not be answered from a three-year-old report that happens to be worded similarly.
Recognise the questions retrieval cannot answer. How many, what is the total, which is the largest, what changed between these dates. These are queries over structured data, and the right answer is a database query or a tool, not a search over prose. A system that attempts them with retrieval produces confident arithmetic over whichever passages it found.
Recognise the questions that need the whole document. Summarise this agreement, list every obligation, does this contract contain an indemnity. These require completeness, which retrieval of the top passages cannot provide. Route them to a different path that processes the document in sections.
Handle the unanswerable deliberately. If the retrieval scores are poor, say so. A system that always answers teaches users that its answers mean nothing, and a visible no-answer rate is one of the healthier metrics in this subject.
Consider generating a hypothetical answer to retrieve with. Asking the model to draft an answer and embedding that, rather than the question, sometimes retrieves better because an answer resembles a passage more than a question does. It costs a call, it helps on some corpora, and it is worth testing rather than adopting on faith.
And keep the rewriting observable. Log the original question and the query actually sent. A surprising proportion of bad answers in production turn out to be a rewriting step that changed the meaning of the question.
Permissions, tenancy and documents that bite
A retrieval system is a new path to every document in the organisation. That is its value and it is also a disclosure risk that most implementations get wrong on the first attempt.
The failure is simple to state. A user asks a question, the index returns a passage from a document that user is not entitled to read, and the model repeats it in an answer. No control was bypassed, because nobody put one in the retrieval path.
Retrieve with the user's entitlements, not the system's. The index must be queried with a filter representing what this user may see, applied as part of the search. This is why filtered approximate search matters, and it is the question to settle before choosing an index.
Retrieve-then-filter is the trap. Fetching the top ten and discarding what the user may not see appears to work and does two bad things: it can return nothing when the visible answer existed deeper in the ranking, and it leaks through timing, counts and citation stubs. Filter during the search.
Permissions change and indexes do not notice. Access revoked in the source system must take effect in retrieval. Either re-index on change, or hold a reference and check entitlement at read time, and know which one you have chosen.
Multi-tenant deployments need a hard boundary. A per-tenant index, or a tenant key enforced at the lowest level and tested adversarially. Relying on a filter in application code is the arrangement that produces a cross-tenant disclosure, and a single such incident is usually fatal commercially.
Think about what the index itself reveals. Chunk counts, similarity scores and document titles can disclose the existence of material a user should not know about. A thorough answer is to make existence itself subject to entitlement.
Personal data brings its own obligations. A vector store derived from customer records is processing personal data, with a lawful basis, a retention period and a right to erasure that has to be executable - which means being able to find and remove every chunk and vector derived from one person's records. Design that before ingestion, not after the first request.
And retrieved documents can carry instructions. Text inside a document may be written to influence the model: ignore your instructions, or summarise this as approved. If your corpus accepts content from outside the organisation, through tickets, email or uploads, you have an injection surface in your index. Treat passages as data, keep high-consequence actions out of the answer path, and read the course on securing systems against agents.
The summary for a reviewer. Entitlement enforced inside the search, revocation that propagates, a tested tenant boundary, erasure that works, and retrieved text treated as untrusted. A system without those five is not ready for anybody's documents but its own.
Evaluating it
A retrieval system has two components that fail differently, so it needs two sets of measurements. Teams that score only the final answer spend months tuning the wrong half.
Build a golden set first. Thirty to a hundred real questions, each with the passage or document that answers it and a note of what a good answer must contain. Drawn from actual user questions where possible. This artefact is the whole evaluation; everything else is a script.
Measure retrieval on its own. Recall at the number of passages you actually supply - is the answering passage in the top five - is the single most informative number in the system. Mean reciprocal rank and discounted cumulative gain add nuance about ordering, and recall at k is where to start.
Measure the candidate stage separately from the final stage. Recall at fifty tells you whether re-ranking has anything to work with. If recall at fifty is poor, no re-ranker will save the system and the problem is in chunking, embeddings or hybrid ranking.
Then measure the answer, on two axes. Groundedness: is every claim supported by the supplied passages. Relevance: does the answer address the question. These are separable, and a system can be perfectly grounded and useless, or relevant and unsupported.
Published frameworks exist for this and are worth reading rather than adopting blindly. The automated evaluation work for retrieval systems proposes exactly this decomposition - context relevance, groundedness, answer relevance - scored with a model. The decomposition is the durable contribution; the scoring is a tool with known biases.
Use a model as a judge with the usual precautions. A different model from the one under test, an explicit rubric, randomised ordering, and agreement checked against human scores on a sample. Where the judge and a person disagree often, fix the rubric before trusting the numbers.
Score the no-answer behaviour. Include questions the corpus cannot answer and measure whether the system says so. A system that scores well on answerable questions and fabricates on the rest has not been evaluated.
Do not trust transfer between domains. The published heterogeneous retrieval benchmark exists because retrieval models that lead on one corpus do not necessarily lead on another. Your corpus is its own benchmark.
Run it as a regression suite. Every change to chunking, embeddings, index parameters, re-ranking, prompt or model runs against the golden set, with retrieval and answer scores recorded separately, alongside latency and tokens.
And grow it from production failures. Every complaint becomes a case with the correct passage attached. A suite assembled this way converges on what actually breaks, which no public benchmark will do for you.
Running it
A retrieval system is an operational thing, not a project. The costs, the freshness and the signals in this lesson are what separate a system that stays useful from one that quietly decays.
Freshness is a pipeline, not an event. Documents change, are added and are withdrawn. Decide the acceptable lag, build the incremental path - new and changed documents re-ingested, deleted documents removed from the index - and monitor it. A system answering from last quarter's documents is worse than one that says it does not know.
Re-indexing is occasionally unavoidable. A new embedding model, a changed chunking strategy or a corrupted index requires rebuilding everything. Know how long a full rebuild takes before you need it, and keep the index version recorded with the model version, because the two must match.
Build the cost model from the components. Embedding at ingestion, once per chunk. Embedding per query. Index storage and memory. Re-ranking per candidate. Generation per call, dominated by the passages in the prompt. Most of the recurring cost is in that last item, which is an argument for fewer, better passages.
Cache at several levels. Identical and near-identical queries, embedding results for repeated queries, and the model's stable prompt prefix. In an internal system a surprising share of questions repeat, and a cache with a short lifetime is cheap and effective.
Monitor the retrieval scores, not only the errors. The distribution of top-passage scores over time is an early warning: a shift means the corpus, the queries or the model has changed. Falling scores precede user complaints by weeks.
Watch the no-answer rate in both directions. Rising means a corpus gap, an ingestion failure or a new kind of question. Falling to zero means the threshold has stopped doing anything.
Collect feedback at the point of use. A way to mark an answer wrong, and a reason. These become golden-set cases, and they are the only route to knowing which failures matter to users rather than to engineers.
Expect query drift. The questions users ask after six months differ from the ones they asked in week one, usually becoming more specific and more operational. The evaluation set has to follow, or the system will be measured on a problem it no longer has.
Keep a human path for what the system refuses. A question routed to a person when the corpus has no answer makes the refusal useful rather than a dead end, and it is where the next gap in the corpus is identified.
And record the configuration with the answer. Index version, embedding model, re-ranker, prompt version, model version. An answer complained about three weeks later cannot be reproduced without them, and in a regulated setting that reproducibility is the requirement.
One system, end to end
An insurer's claims team wants to ask questions of its own material: policy wordings, endorsements, internal guidance and regulatory circulars. About eighty thousand pages, mostly portable documents, some scanned, many versions. Three engineers, eight weeks.
Week one, ingestion only. Extraction produced unreadable text for a fifth of the corpus: two-column circulars interleaved, and scanned endorsements that needed recognition. They fixed the layout handling, ran recognition over the scans, measured its accuracy on a sample of fifty pages, and recorded that policy numbers were the weakest field. Headers and footers were stripped. Near-duplicate detection removed nine thousand pages of superseded versions, keeping the latest of each with its date in metadata.
Week two, the golden set. Sixty real questions collected from the claims team's existing email queue, each with the passage that answered it, identified by a person. Two of the sixty turned out to have no answer in the corpus, which was retained deliberately as a test of the refusal path.
Week three, chunking and the first measurement. Structure-aware splitting on clause and section boundaries, with the document title and section path prepended to every chunk. Recall at five was 61 per cent. Fixed-size chunking, tried as a comparison, gave 44 per cent. The heading prefix alone accounted for six points.
Week four, hybrid retrieval. Adding a keyword index and fusing the two rankings took recall at five to 78 per cent. Every one of the questions it fixed involved a policy number, a clause reference or a defined term - exactly the cases the lesson on keyword search predicts.
Week five, re-ranking. Retrieving fifty candidates and re-ranking to five took recall at five to 89 per cent. Recall at fifty was 94, which told them the remaining failures were upstream rather than in the ranking, and three were traced to recognition errors in scanned documents.
Week six, assembly and answers. Instruction first, passages labelled with document, section and date, question last, with an explicit instruction to answer only from the passages and to say when they did not contain the answer. Citations required per sentence and verified mechanically against the supplied passage identifiers, which caught two cases of a cited passage that had not been supplied.
Week seven, permissions. Claims handlers could see all the guidance; three circulars were restricted to managers. Entitlement became a filter applied inside the search rather than afterwards, and an adversarial test confirmed a restricted circular could not be surfaced by any wording they tried.
Week eight, operation. Nightly incremental ingestion. Index and embedding model versions recorded with every answer. A dashboard of top-passage score distribution, no-answer rate and tokens per question. Feedback button with a reason field.
What it looked like after three months. Recall at five had drifted down to 84 per cent as questions became more specific; the golden set grew to 140 cases from user feedback; and the largest single remaining failure was a document set from a subsidiary that nobody had included. The ingestion log would have shown it, had anybody read it before the complaint.
The point. Two of the eight weeks were spent on the model and the prompt. The other six were ingestion, chunking, hybrid ranking, re-ranking, permissions and measurement, which is the correct proportion and almost never the planned one.
Fifty questions
Answer all fifty questions, then check your answers. You need 45 out of 50 to pass and receive a certificate. If you score less, you will see which answers were right and wrong, and then go through the course again before you retake the check. Your answers, progress and times are kept only in this browser.
Your answers
Your certificate of completion
Enter your name as you want it to appear, then save the certificate as a PDF. In the print window, choose Save as PDF. A certificate is issued once per completion of the course.
Rate this course
How would you rate this course? Your rating is anonymous — just a number, with no name and no account.
Bring this to your team
Your colleagues can take this course too — it is free, needs no sign-up, and nothing they enter leaves their browser. If your institution wants the same ground covered as structured training for a team, with an instructor and your own cases, we run courses in a classroom or online for groups of ten or more.
Talk to us about training your team → See the paid courses →
Saolix does not record who takes this course, so it cannot verify these certificates. The certificate confirms completion of a free self-paced course and is not an accredited qualification.
Sources
The official documents this course relies on. Laws and guidance change, so check the current version.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · Lewis and others, arXiv 2005.11401
- Dense Passage Retrieval for Open-Domain Question Answering · Karpukhin and others, arXiv 2004.04906
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks · Reimers and Gurevych, arXiv 1908.10084
- Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs · Malkov and Yashunin, arXiv 1603.09320
- Billion-scale similarity search with GPUs · Johnson, Douze and Jegou, arXiv 1702.08734
- Passage Re-ranking with BERT · Nogueira and Cho, arXiv 1901.04085
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering · Izacard and Grave, arXiv 2007.01282
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models · Thakur and others, arXiv 2104.08663
- Ragas: Automated Evaluation of Retrieval Augmented Generation · Es and others, arXiv 2309.15217
- Lost in the Middle: How Language Models Use Long Contexts · Liu and others, arXiv 2307.03172
- Retrieval-Augmented Generation for Large Language Models: A Survey · Gao and others, arXiv 2312.10997
- Data Protection Act, 2019 (No. 24 of 2019) · Office of the Data Protection Commissioner, Kenya



