Retrieval-Augmented Generation (RAG)

Imagine you ship a chatbot for your company.

Behind it runs a large language model (LLM). A user asks:

And the bot answers with confident, completely wrong information.

Why? Because an LLM's knowledge is frozen at training time and baked into its weights — this is called parametric memory, meaning "facts the model memorized during training, stored as numbers in the network."

The model never saw last week's update. It cannot "remember" what it was never taught.

That single fact creates three production failures:

• Staleness — the model cannot know about yesterday's policy change, this quarter's prices, or a document you wrote an hour ago.
• Hallucination — when the model does not know, it often invents something plausible instead of saying "I don't know." It makes up case numbers, drug dosages, legal clauses.
• No access control — a model's weights cannot answer "only show me documents this user is allowed to read." Everything it memorized is available to everyone.

So the question becomes

Yes. That is what RAG is for.

RAG — Retrieval-Augmented Generation — is simply

It was introduced by Lewis et al. (2020), and it changes the model's job:

• Without RAG: "recall what you memorized." (Unreliable — memory is frozen and lossy.)
• With RAG: "summarize what you were just handed." (Reliable — models are far better at reading-and-summarizing than at free recall.)

The word breakdown makes it obvious:

• Retrieval = a search system fetches a few relevant text passages from your documents.
• Augmented = those passages are pasted into the prompt, alongside the user question.
• Generation = the LLM writes the answer conditioned on the pasted evidence.

And it composes with fine-tuning (further training the model on your own examples so it copies a style or format). RAG and fine-tuning are not enemies: retrieval supplies the facts, the model supplies the language and reasoning.

The classic trade-off versus fine-tuning:

• RAG updates knowledge by re-indexing documents — minutes, fully auditable, and you can see exactly which passage was used.
• Fine-tuning updates knowledge by retraining weights — hours, opaque, and you cannot point at where the fact lives.

Bonus: RAG gives you citations for free — the answer just points at the retrieved chunks ("per [2]").

Suppose your knowledge base holds just three passages ("chunks") of HR policy.

A user asks:

Step 1 — index (done once, offline): each chunk is turned into a vector (a list of numbers capturing its meaning — topic 162) and stored.

code
Copy
[0] "Dental coverage is 80% after the deductible."
[1] "New parents get 10 weeks of paid leave..."
[2] "The office cafeteria closes at 3 PM."

Step 2 — retrieve (per question): the question is also turned into a vector, and the index scores every chunk by cosine similarity (a 0-to-1-ish "how close in meaning" number).

code
Copy
scores:  chunk[1] = 0.82   ← winner
          chunk[0] = 0.31
          chunk[2] = 0.24

With k = 2, chunks [1] and [0] are handed to the model.

Step 3 — generate: the prompt is assembled as evidence + question + rules:

code
Copy
Answer ONLY using the numbered context.
Cite sources like [1]. If context is
insufficient, say so.

CONTEXT:
[1] New parents get 10 weeks of paid leave...
[0] Dental coverage is 80% after the deductible.

QUESTION: How many paternity leave days do I get?

The model now reads chunk [1] and answers "10 weeks ([1])" — a fact it never memorized.

Notice the safety valve: if the top score had been 0.18 — nothing really matching — a production system refuses or escalates instead of answering. That gate ("max similarity below threshold → refuse") blocks a whole class of hallucinations before they happen.

RAG is really two separate flows that meet at the index.

code
Copy
OFFLINE (slow, thorough — runs when docs change)

  PDFs / Wiki / Code → chunk → embed → ┌─────────────┐
                                        │ Vector Index│
                                        └─────────────┘
ONLINE (fast — runs on every question)      ▲
                                            │ top-K lookup
  Question → embed → similarity search ─────┘
       → assemble context → LLM → grounded answer + citations

Two things this picture makes obvious:

• The index is built once and reused by every query — that is why updating knowledge means re-uploading documents, not touching the model.
• The answer can only ever be as good as what the lookup returned. Retrieval is the ceiling; generation just works under it.

The model itself never "reads" your whole corpus. It only ever sees the 2–8 small chunks the search handed it — a book open at the right page.

Carry one analogy through everything below: RAG is an open-book exam.

The LLM is a brilliant student. Without RAG, you force her to answer every question from memory — and when her memory runs out (the training cutoff), she guesses politely instead of blanking the page. That is a hallucination.

With RAG, the rules change:

• The index is the reference book, tabbed and organized so the right page can be found fast (chunking, embedding, vector storage).
• The retriever is the assistant who, for each question, flips to the 4 most relevant tabs and puts those pages in front of her.
• The prompt is the exam instruction: "answer only from these pages, and cite the page number."
• The answer comes with citations, because she literally had the pages in hand.

Now the fine-tuning comparison makes sense too. Fine-tuning is sending the student to a new school so she memorizes the updated edition — slow, and nobody can check what she actually learned. Re-indexing RAG is just swapping in the new edition of the book — minutes, and you can open to the exact page.

And the "answerability gate" is the student saying: "these pages do not cover question 7" — instead of inventing an answer.

Everything else in this topic — hybrid search, rerankers, agentic loops — is about one thing: making the assistant flip to better pages, faster.

Indexing (offline). Documents are parsed, cleaned, chunked (how to slice them is topic 166), embedded into vectors (topic 162), and stored in a vector database (topic 163) along with metadata — source, ACL tags (access-control labels), timestamps — so retrieval can filter on them later.

Retrieval (online). The question is embedded with the same model that embedded the documents, and an approximate nearest-neighbor (ANN — "find nearly-closest vectors without checking all of them") search returns candidate chunks. Production systems rarely stop at raw top-K: they rewrite the query, fuse dense (meaning) with sparse (exact-word) results (topic 165), rerank with a more precise model (topic 167), and pack the surviving chunks under a token budget.

Generation. The prompt template orders instructions, the question, and the evidence blocks — each chunk labeled by source and score so the model can cite. The model is instructed to answer only from context and to say "I don't know" when evidence is insufficient. Position matters too: models attend best to the beginning and end of long contexts — the "lost in the middle" effect (Liu et al. 2023) — so the highest-confidence chunks go first or last.

The core loop is small enough to write in about twenty lines: embed → top-K by cosine → paste into a prompt. Everything else in RAG engineering is quality control wrapped around this skeleton.

Read the code below with the open-book analogy in mind: top is the assistant picking the pages, context is the open book, and the prompt is the exam instruction.

import numpy as np

def rag_answer(question, chunks, embedder, llm, k=4):
    # 1) Index (here: in-memory; in prod: a vector DB)
    vecs = np.array([embedder(c.text) for c in chunks])
    vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)

    # 2) Retrieve
    qv = embedder(question); qv /= np.linalg.norm(qv)
    scores = vecs @ qv
    top = [chunks[i] for i in np.argsort(-scores)[:k]]

    # 3) Generate with grounded citation instructions
    context = "\n\n".join(f"[{i}] ({c.source}) {c.text}"
                           for i, c in enumerate(top))
    prompt = (f"Answer ONLY using the numbered context. "
              f"Cite sources like [2]. If the context is "
              f"insufficient, say so.\n\nCONTEXT:\n{context}"
              f"\n\nQUESTION: {question}")
    return llm(prompt), [c.source for c in top]

When a RAG answer is wrong, the error lives in exactly one of two stages — so you need separate metrics for each.

• Retrieval failures (wrong context): the answer is perfectly faithful to chunks that simply do not contain the answer. The student copied the wrong page. Metrics: recall@k (did the right chunk appear in the top-k at all?), MRR (how high did it rank?), nDCG (ranking quality with position weight) against a labeled set of query→correct-chunk pairs.
• Generation failures (unfaithful answer): the correct context was retrieved, but the model ignores it or contradicts it. Metrics: faithfulness / groundedness — an LLM judge checks whether each claim in the answer is supported by the retrieved chunks (tools like RAGAS, TruLens do this).

Once naive RAG works, modern variants attack specific weaknesses:

• Self-RAG trains the model to emit reflection tokens that decide when to retrieve and whether the retrieved passages are actually relevant.
• Corrective RAG (CRAG) routes low-confidence retrievals to web search instead of trusting a bad index hit.
• Agentic RAG lets the model loop — decompose a hard question into sub-queries, retrieve iteratively, and stop when evidence is sufficient (topics 169–170: agents and their loops).