A compliance officer asks a simple question: what does our policy say about this? A language model on its own can only answer from what it saw in training, which ends at a date and never included your policy manual. Retrieval-augmented generation is the usual fix. It is useful, it is checkable, and it is not as safe as it looks.

The problem RAG was built to solve

A large language model writes its answer one small piece of text at a time. Each piece, called a token, is chosen because it is likely to come next. The model has no list of checked facts to consult. It has patterns learned from a large body of training text, and that text has a cut-off date. Most current models are built on the transformer, a design set out in a 2017 research paper, which lets the model weigh how each part of a text relates to the rest.

Two gaps follow for any regulated firm. The model does not know about anything that happened after its training ended. And it has never seen your own material: your procedures, your risk assessment, your board-approved policies, the circular your regulator sent last month.

Retrieval-augmented generation (RAG) closes those gaps without retraining the model. The term comes from a 2020 research paper by Lewis and colleagues. The idea is simple. Before the model answers, the system looks up relevant passages from a chosen set of documents and hands them to the model alongside the question.

How a RAG system works

There are two stages, and it helps to keep them apart.

Preparation happens once, and again whenever documents change. The documents are cut into short passages, usually called chunks. Each chunk is converted into an embedding: a list of numbers that captures what the passage means. The embeddings go into a searchable index, often called a vector database.

Answering happens for every question. The question is converted into an embedding in the same way. The system finds the chunks whose meaning sits closest to the question and places them in the prompt. The model is told to answer only from those passages and to cite them.

Many systems run an ordinary keyword search as well, an approach called hybrid search. This matters more in compliance than in most fields. Meaning-based search is good at ideas, but it can miss exact items: a customer name, an account number, a product code, a specific section reference. Keyword search catches those.

Five steps in a retrieval-augmented generation system: documents are split into chunks, the chunks are stored in an index as embeddings and keywords, the passages closest to a question are retrieved, and the model writes an answer that cites those passages. PREPARE ONCE, THEN ANSWER EACH QUESTION Documentspolicies · procedures Chunksshort passages Indexmeaning + keywords Retrieveclosest passages Answercites its passages IF RETRIEVAL PICKS THE WRONG PASSAGES, THE ANSWER WILL BE WRONG
The first three steps build the library. The last two run for every question. The quality of the answer depends on the quality of what is retrieved.

Why grounding and citations matter in regulated work

A plain language model gives you an answer and asks you to trust it. A RAG system gives you an answer and the passages it was drawn from. For a compliance team, that difference is the whole point.

Answers can be checked. A reviewer can open the cited passage and see whether it says what the answer claims. In regulated work, an answer nobody can trace is an answer nobody can defend to an auditor or a supervisor.

Answers can stay current. When a policy is updated, the index is refreshed. The model itself does not need to be retrained. The answer reflects the version of the document that is in the library today.

Access can follow the rules you already have. Retrieval can be limited to the documents a person is allowed to see. A branch officer need not get answers drawn from board papers or investigation files.

Where it still fails

RAG reduces made-up answers. It does not remove them. The failures are worth knowing by name, because each needs a different control.

  • Wrong retrieval. If the search returns the wrong passages, or misses the right one, the model will answer confidently from the wrong material. Exact references are a common weak spot when only meaning-based search is used.
  • Misreading the source. The model can be given the correct passage and still misstate it. A citation shows where the answer came from, not that the answer is faithful to it.
  • Hallucination. Models can produce fluent, assured statements that are simply untrue, with or without retrieved documents.
  • Variability. The same question can produce a different answer, especially after a change of model, prompt or settings.
  • Prompt injection. Text hidden inside a retrieved document or web page can carry instructions that take over the model's behaviour. The risk grows when the system can use tools or take actions.
  • Bias carried over from the data the model was trained on.
  • Data exposure. Text sent to an outside AI service leaves your control, including the retrieved passages that travel with each question.
A citation tells you where to look. It does not tell you the answer is right. Someone still has to look.

Advise, do not decide

The safer pattern for regulated work is to give the model an advisory role. It can summarise a procedure, explain a clause, draft a first note or suggest which policy applies. Fixed rules and people make the decisions. The more a system is allowed to do on its own, the tighter its permissions should be, the more fully its actions should be recorded, and the more often a person should approve what matters.

That pattern sits comfortably with the way risk teams already work. For wider reading on managing these risks, the US National Institute of Standards and Technology's AI Risk Management Framework and the OWASP Top 10 for LLM Applications are useful public references.

Questions to ask before you trust it

Whether you are building a RAG system or buying one, six questions show whether it is fit for regulated use:

  1. Which documents are in the library, who owns them, and how quickly does an updated policy reach the index?
  2. Does it combine keyword and meaning-based search, so that names, account numbers and section references are found reliably?
  3. Does every answer cite its passages, and does anyone actually check them?
  4. Are access rules applied at retrieval, so people only get answers from documents they may see?
  5. Has it been tested on our own cases, and do we measure how often its suggestions are right before relying on them?
  6. Do we record the model version and settings, so that a change in answers can be explained later?

In Kenya, personal data in a prompt or a retrieved passage is covered by the Data Protection Act, 2019. Sending that text to an outside AI service can also engage the Act's rules on transferring personal data outside Kenya. Map where the data goes before it goes there.

Our free course LLMs, SLMs, Generative AI and RAG covers how language models work, when a small model fits, how RAG grounds answers in your documents and how to use these systems safely, with a knowledge check and a certificate.

Sources