What RAG is: letting a model read your documents

A language model has read half the internet, but it has not read your things: the company handbook, your course notes, the twenty PDFs you downloaded and never opened again. That knowledge is frozen at training time, and none of it is yours.

RAG — retrieval augmented generation — solves the problem in the simplest way available: search your documents first, then answer using what was found. It is the difference between a closed-book exam and an open-book one. The open-book student does not know more: they know where to look.

The problem, plainly

You: "What does chapter 5 of my handbook say?"
LLM: "I don't have access to your documents. I can explain the general concepts..."

You: "What is the holiday procedure at my company?"
LLM: "Companies generally provide..."   ← generic, which means useless

The model can know everything about employment law and nothing about your policy. And when it does not know, it often does not stay silent: it fills the gap with something plausible. Anchoring answers to real documents is the most effective way to curb that.

The three phases

RAG splits into one phase you run once and two that repeat with every question.

1. Indexing — once. Documents are read, split into chunks, turned into vectors and stored.

PDFs, notes, pages
      │
      ▼
[LOADING]     reads PDF, Markdown, DOCX, HTML
      │
      ▼
[CHUNKING]    splits into chunks of a few hundred words
      │             (the most delicate choice in the whole pipeline)
      ▼
[EMBEDDING]   each chunk becomes a list of numbers standing for
      │             its meaning
      ▼
[STORE]       the vectors land in a database built to search by
                  similarity

2. Retrieval — every question. The question is turned into a vector with the same model used for the documents, and the closest chunks are pulled out.

"How does expense reimbursement work?"
      │
      ▼
[EMBED THE QUESTION]   same space as the documents
      │
      ▼
[SIMILARITY SEARCH]    which chunks point in the same direction?
      │
      ▼
[TOP K RESULTS]        the 3-5 most relevant chunks

3. Generation. The retrieved chunks go into the prompt alongside the question, and the model answers by reading them.

SYSTEM:   Answer only from the context below.
          If the answer is not there, say so.

CONTEXT:  [chunk 1: "Expenses must be reported within..."]
          [chunk 2: "The daily cap is..."]

QUESTION: How does expense reimbursement work?

The model has learned nothing new: it is reading. The knowledge lives in the store, not in its weights. That single point explains almost every RAG behaviour, good and bad.

Why Ctrl+F is not enough

Classic search looks for words. RAG looks for meaning, because it works on embeddings: texts that mean the same thing land close together in space even with no vocabulary in common.

You search: "polymorphism"

Ctrl+F  finds:     "Polymorphism is a concept..."
        misses:    "the ability of an object to take different forms"

RAG     finds:     "Polymorphism is a concept..."
        finds:     "the ability of an object to take different forms"
        finds:     "override and overloading are the two mechanisms..."

The reverse is also true, and it is a real limitation: for a product code, an acronym or a proper name, keyword search is more precise than semantic search. That is why serious systems run both and merge the results.

The parts of a RAG system

Component	What it does	Examples
Loader	Reads formats and extracts the text	PyMuPDF, Unstructured
Splitter	Breaks documents into chunks	recursive splitters, structure-aware ones
Embedding model	Turns text into vectors	nomic-embed-text, BGE-M3
Vector database	Stores and searches by similarity	Chroma, FAISS, Qdrant
Retriever	Picks the chunks to hand to the model	similarity search, reranking
Language model	Writes the answer from those chunks	the one you already run locally

Five of those six parts are not the model. It is the most useful thing to grasp before starting: the quality of a RAG system is decided almost entirely outside the LLM.

When it helps and when it is overhead

Situation	Does RAG help?	Why
One chapter you want to understand	No	Paste it into the prompt
Twenty PDFs to search a concept in	Yes	Nobody re-reads those by hand
Documentation that changes weekly	Yes	You update the store, not the model
Comparing how three sources explain one thing	Yes	It retrieves the parallel passages
Giving the model a fixed tone or format	No	That is a system prompt job

The first row is the one people get wrong most often. With today’s context windows a thirty-page document fits whole: building a retrieval pipeline around it is extra work that makes the result worse. RAG starts to pay off when the documents no longer fit, or when they keep changing.

RAG or fine-tuning?

They answer two different questions and get confused constantly.

	RAG	Fine-tuning
What changes	What the model reads	How the model behaves
Knowledge	Updatable in a second	Frozen into the weights
Cost of an update	Add a file to the store	Retrain
Citable sources	Yes	No
Good for	Facts, documents, procedures	Tone, format, domain jargon

Rule of thumb: if you need the model to know something, RAG. If you need it to answer a certain way, fine-tuning. Serious setups use both, but almost nobody should start with the second.

Where it breaks

RAG fails in the same few places every time, and none of them is the model’s fault.

Confusing documents stay confusing. If the original handbook is badly written, retrieval returns badly written chunks. No model compensates for a poor source.

Wrong chunking. Chunks that are too small lose their context and become meaningless; chunks that are too large blend too many topics and end up close to no question in particular. It is mistake number one, and it is the subject of the next lesson.

Two different embedding models. Index with one model, query with another, and the vectors live in incompatible spaces: the results are noise. Change the model and you rebuild the store from scratch.

Too many chunks in the prompt. Passing fifteen results “to be safe” saturates the context and degrades the answer: the model gets distracted by mediocre passages. Three relevant chunks beat fifteen decent ones, every time.

False positives. The retriever always returns something: even when the answer is not in your documents, it hands over the least distant chunks. Without an explicit instruction to say “I don’t know”, the model builds an answer on top of them.

The illusion of learning. After a good session it feels like the model now “knows” your documents. It does not: the next question starts from zero and queries the store again. RAG teaches nobody anything.

Two ways to start

Route 1 — ready-made applications
  A local interface with RAG built in: load documents, ask questions.
  Half an hour of work, limited control over how texts are split.

Route 2 — Python frameworks
  You assemble the pipeline piece by piece.
  A few hours and some code, control over every parameter.

To study or search your own documents, the first route is plenty: an interface like Open WebUI on top of Ollama does RAG without a line of code. The second is for when retrieval has to be measured and tuned — and that is where you find out how much weight the ready-made app carried for you.

In short

Concept	In one line
RAG	Search your documents first, then answer from those chunks
Analogy	An open-book exam instead of a closed-book one
Indexing	Load, split, embed, store: once
Retrieval	The question becomes a vector and pulls the closest chunks
It is not	Teaching the model: the knowledge stays in the store
When to skip it	If the document fits the context window, paste it
Typical mistake	Bad chunking and too many chunks passed just in case
Iron rule	Same embedding model for indexing and for querying