RAG, explained: how an AI answers from your documents

    “Chat with your documents” is one of the most common things companies ask for. The technique behind it is called retrieval-augmented generation, or RAG. Here is how it works, using a made-up example: a company leave policy stored as a long PDF.

    The idea in one sentence

    A language model does not know your documents, so before it answers, the system finds the relevant passages and hands them to the model along with the question.

    Two stages

    Figure 1 · RAG in two stages

    DocumentsSplit into chunksIndex for search
    QuestionRetrievefind matching chunksModelanswers from themAnswerwith sources
    Top row: preparing the documents once. Bottom row: what happens on every question.
    1. Prepare. The policy is cut into chunks, such as a few paragraphs each. Each chunk is turned into a list of numbers, called an embedding, that captures its meaning, and stored in an index.
    2. Retrieve. A user asks, “How many days of leave do new joiners get?” The question is turned into numbers the same way, and the system finds the chunks whose meaning is closest.
    3. Generate. The model gets the question and those chunks, and writes an answer based on them. A good system shows which chunks it used.

    Where it goes wrong

    • The right passage was not found. The model then answers from the wrong text, or from nothing.
    • The chunk lacks context. A paragraph that says “this does not apply to contractors” is useless without the heading above it.
    • The model ignores or over-reads the text. It can add details that are not in the source.
    • The document is out of date. RAG repeats what is in the index, old or new.

    How to check one

    Treat retrieval and answering as two things to test. First, for a set of real questions, check whether the right passage was retrieved. Then check whether the answer matches it. When something fails, you will know which stage to fix. Always show sources in the answer, so a person can verify.

    Building this for your own documents is one of the project types we take on. See our services page.

    Want help putting this into practice?

    Book a free consultation and tell us about your team or school.

    Book a free consultation