RAG, explained: how an AI answers from your documents
“Chat with your documents” is one of the most common things companies ask for. The technique behind it is called retrieval-augmented generation, or RAG. Here is how it works, using a made-up example: a company leave policy stored as a long PDF.
The idea in one sentence
A language model does not know your documents, so before it answers, the system finds the relevant passages and hands them to the model along with the question.
Two stages
Figure 1 · RAG in two stages
- Prepare. The policy is cut into chunks, such as a few paragraphs each. Each chunk is turned into a list of numbers, called an embedding, that captures its meaning, and stored in an index.
- Retrieve. A user asks, “How many days of leave do new joiners get?” The question is turned into numbers the same way, and the system finds the chunks whose meaning is closest.
- Generate. The model gets the question and those chunks, and writes an answer based on them. A good system shows which chunks it used.
Where it goes wrong
- The right passage was not found. The model then answers from the wrong text, or from nothing.
- The chunk lacks context. A paragraph that says “this does not apply to contractors” is useless without the heading above it.
- The model ignores or over-reads the text. It can add details that are not in the source.
- The document is out of date. RAG repeats what is in the index, old or new.
How to check one
Treat retrieval and answering as two things to test. First, for a set of real questions, check whether the right passage was retrieved. Then check whether the answer matches it. When something fails, you will know which stage to fix. Always show sources in the answer, so a person can verify.
Building this for your own documents is one of the project types we take on. See our services page.