RAG or fine-tuning: the 80/15/5 rule

The video is in Italian; this page is the written version in English.

The model answers with confidence, and it is wrong: it has never read the latest quarterly report. From here on, every AI project meets the same fork in the road. Do you hand the model the documents at the moment it has to answer (RAG), or do you retrain it so that it makes them its own (fine-tuning)? The two roads solve different problems, cost different money and fail in different ways. This lesson is about choosing without anxiety.

The open-book exam and the all-nighter

The student with the book open on the desk has access to all the information but has not internalised any of it. If the professor switches to a new edition at the last minute, no problem: flip through the new text and the answers change instantly. That is RAG: the documents are retrieved and passed to the model inside the prompt, every time, and the model itself never changes.

The student who spent months studying no longer needs the book: the subject has become part of how they reason. But if the facts change, updating on the fly is impossible without going back to the desk. That is fine-tuning: a training run that permanently changes the model’s weights.

There is also the well-prepared student who still brings their notes: the best of both worlds, and the most expensive. We will get to it, because it is the right combination more often than it looks.

Data versus behaviour

Metaphors aside, the real difference fits in one word: RAG acts on data, fine-tuning on behaviour.

Aspect	RAG	Fine-tuning
What it does	Adds temporary context to the prompt	Changes the model’s weights
Does the model change?	No, it stays identical	Yes, permanently
Setup cost	Low: vector DB + embedding model	High: GPU, dataset, training
Cost per request	Medium: embedding + search + generation	Low: generation only
Setup time	Hours (index the documents)	Days or weeks
Updating	Add or remove documents	Retrain
Citing the source	Yes, down to the exact passage	No: the knowledge is fused into the weights
Style and tone	Unchanged	Can change
Hallucinations	Reduced: the answer is anchored to sources	Possible, even on the trained domain
Privacy	Documents in a database you control	Data “baked” into the model
Needs a GPU	No	Yes, for training

One row of this table matters more than the others: citing the source. If anyone will ever ask “where does this answer come from?”, fine-tuning cannot tell them. RAG can.

The 80/15/5 rule

In practice, the cases split roughly like this.

80% of projects are solved by RAG. Company knowledge bases, manuals, procedures, technical documentation that changes, regulations, price lists, your own archive of notes. The model already knows how to reason, summarise and compare: it only lacks the data, and the data changes. A typical example is the company leave policy: RAG searches the knowledge base, the model answers citing the document, and when the policy changes you replace the file.

15% need both. An assistant with a very specific tone that must nevertheless read fresh data; a “domain expert” that reasons like a doctor and consults the latest guidelines; a system that has to respect a strict output format and rely on current documents. The division of labour is clean: fine-tuning teaches how to answer, RAG supplies what to answer about.

5% is pure fine-tuning. Radically changing the writing style (a model that writes in a dialect), a repetitive specialised task (classifying thousands of tickets with the company’s internal categories), a poorly supported language, a proprietary programming language. This is not about giving more information: the model has to think differently.

The decision tree, in two questions

Does the model already answer well with a good prompt?
├── YES → Is the problem that it lacks the right DATA?
│         ├── YES → RAG
│         └── NO  → Nothing needed: improve the prompt
└── NO  → Is the problem the BEHAVIOUR (tone, format, jargon)?
          ├── YES → Do you have a quality dataset?
          │         ├── YES → Fine-tuning
          │         └── NO  → Build the dataset first
          └── NO  → Do you need both behaviour and data?
                    └── YES → Both (fine-tuning + RAG)

The two questions that matter:

1. Is the model missing data? It answers confidently but wrongly because it has never seen the right document. A data problem: RAG.
2. Does the model know the facts but refuse to write them the way you need? It ignores the format, drifts in tone, does not use the field’s jargon, and no prompt convinces it. A behaviour problem: fine-tuning, provided you have a curated dataset.

Before answering the second one, always exhaust prompt engineering first: it is free, immediate, and solves more cases than people expect.

What it really costs

Strategies look great on paper; then there are budgets.

RAG, locally. A free embedding model (with Ollama, for instance nomic-embed-text), a free vector database (ChromaDB), a thousand pages indexed in about half an hour on a CPU, a hundred megabytes or so for ten thousand documents. Per request: a few milliseconds of embedding and search, then the model’s own time. Locally the total is essentially zero; with a paid API, fractions of a cent per question. If you want to see it assembled piece by piece, the full pipeline has its own lesson.

Fine-tuning. The hurdle is all at the start: a dataset of 500-5,000 quality examples (days or weeks to build), 1-4 GPU hours per training run (in the cloud, roughly 5 to 50 euros per attempt), typically 3 to 10 attempts before a good model, and someone on the team who knows machine learning. In return, once it is ready, each request costs only inference. Heavy setup, light usage: the opposite profile to RAG.

The right order: prompt, then RAG, then fine-tuning

The three levers are not equal alternatives: their costs climb, and they should be tried in order. A concrete example, a customer-support assistant for a company:

1. Prompt          "Answer formally, in Italian, five lines at most,
                    and say so if you don't know."          → zero cost, right now
2. + RAG           Manuals, FAQ and price list indexed; the model
                    cites the page it took the answer from    → hours of work
3. + Fine-tuning   Only if, after 1 and 2, the model still ignores
                    the format or the field's jargon          → weeks and GPUs

Most projects stop at step 2, and not out of laziness: step 3 solves a problem (behaviour) that often is not there. When it is, the combination works well precisely because each part does one thing. The medical assistant of the classic example reasons with clinical caution because it was trained to, and cites the latest guidelines because RAG hands them over: when the guidelines change, you update the database, not the model.

The four most expensive mistakes

Using fine-tuning to teach facts. “I want the model to know everything about the new product line” is the sentence that most often triggers a pointless training run. Changing the weights is a terrible way to make a model memorise details and spec sheets: it will mix them up, and next week they will have changed anyway. For facts and documentation there is one road, RAG.

Using RAG for formatting. The opposite extreme: “I want it to always answer in this exact JSON schema”, and hoping that adding documents to the context will work miracles. RAG brings data; it does not change how the model decides to write. A thousand extra documents will not persuade it to produce perfect JSON. If the prompt is not enough, that is a behaviour problem: fine-tuning.

Believing RAG teaches the model something permanent. It does not modify the model. Once the conversation is closed, the model has forgotten everything: the “memories” live in the vector database, not in the weights. That is a property, not a defect, but you need to know it.

Thinking more data means better results. In fine-tuning, dataset quality matters more than quantity: fifty perfectly curated examples are worth more than five thousand mediocre ones. “Garbage in, garbage out” counts double here, because the garbage ends up in the weights.

In short

Question	Short answer
The model lacks data	RAG: documents in the prompt, model untouched, sources citable
The model knows the facts but won’t write them as needed	Fine-tuning, with a curated dataset
You need a precise style and fresh data	Both: fine-tuning for the how, RAG for the what
How often RAG is enough	About 80% of cases; pure fine-tuning is 5%
Where the cost of fine-tuning sits	All upfront: dataset, GPU, attempts, expertise
The most common mistake	Training a model to learn facts that keep changing