Pre-training, SFT and RLHF

The video is in Italian; this page is the written version in English.

When you download a model and find it labelled base, instruct or chat, you are looking at three different stages of the same journey. The journey has three phases, and each one transforms the model’s behaviour deeply.

1. PRE-TRAINING          2. SFT                  3. RLHF / DPO
"read the internet"      "learn to answer"       "learn to answer well"

trillions of tokens  →   thousands of        →   response pairs with human
of raw text              question/answer pairs   judgement: "A better than B"

BASE model               INSTRUCT model          ALIGNED model

Phase 1 — Pre-training

The model reads trillions of tokens and learns one thing: predict the next token. No questions, no answers, just raw text — books, web pages, code, encyclopaedias.

Almost everything the model knows comes from here: grammar, word meaning, facts about the world, programming syntax, other languages if they were in the data.

It is also incomparably the most expensive phase: months of computation on thousands of GPUs, costs in the millions. That is why there are few base models in the world and thousands of specialised variants of them.

The result is a very well-read parrot that cannot converse: ask “what is the capital of France?” and it might reply by listing more similar questions, because it saw plenty of questionnaires in the data.

Phase 2 — SFT, supervised fine-tuning

Here the model sees for the first time examples of how to answer: thousands of question-answer pairs written or curated by people.

It learns no new facts — those were all in the previous phase. It learns a format: when someone asks something, the right continuation is an answer, not another question. This is where the instruct model comes from.

The amount of data is tiny compared with pre-training: tens of thousands of examples against trillions of tokens. It changes everything anyway.

Phase 3 — Alignment: RLHF, DPO, GRPO

Knowing how to answer is not enough: between two relevant answers, one may be more useful, clearer, safer. This phase teaches the model to prefer the better one.

The classic method is RLHF: human raters are shown pairs of answers and say which they prefer, those judgements train a model that assigns a score, and finally the model is optimised to maximise it.

DPO reaches a similar result while skipping the intermediate scoring model: it uses the preferred pairs directly, and is far simpler and cheaper. GRPO is the variant that made explicit-reasoning models such as DeepSeek R1 possible.

This is also where the model learns to refuse dangerous requests, to admit it does not know, to be less confidently wrong. And where the famous “alignment tax” is paid: a heavily aligned model can feel more cautious and a little less sharp than the raw one.

Why the distinction matters

If the model…	The problem lives in…
does not know a fact	pre-training (fix with RAG, not fine-tuning)
ignores the format you asked for	SFT, or more simply your prompt
answers but unhelpfully	alignment
refuses harmless requests	over-cautious alignment

Remembering which phase a behaviour comes from stops you looking for the fix in the wrong place.

When the model writes the dataset

The bottleneck of the last two phases is not computation, it is data: thousands of well-written examples are needed, and writing them by hand is expensive. Hence a now dominant practice: using a capable model to generate the data for training another.

Large model
     │  generates thousands of question/answer pairs
     ▼
Synthetic dataset → filtered and verified → training a small model

It works surprisingly well, and it is why small models have appeared in recent years with abilities that previously required far larger sizes.

Three concrete cautions. Filtering is not optional: unverified data propagates the generator’s errors into the trained model. Licences matter, because some services forbid using their outputs to train competing models. And there is a risk of impoverishment: training repeatedly on synthetic material, generation after generation, narrows variety. The recipe that holds is mixed: synthetic for volume, human for grounding.

When the judge is another AI

The final phase requires human judgements on thousands of answer pairs: slow and expensive. The variant that took hold replaces the human rater with a model that judges, guided by a written list of principles.

The advantage is obvious: from thousands of human judgements to millions of automated ones. So is the hidden cost: the judge’s biases enter the aligned model, and they are documented — judge models tend to prefer long, well-formatted, confident answers at equal content.

The same pattern spread beyond training: “model as judge” is now a common way to evaluate running systems. Knowing its flaws before trusting it is essential: if the judge rewards verbosity, optimising for that score produces a verbose system.

In short

Phase	What it teaches	Cost
Pre-training	The language and facts about the world	enormous
SFT	To answer instead of continuing	low
RLHF / DPO / GRPO	To prefer the better answer	medium
Synthetic data	One model writes the data to train another: effective, needs filtering
Automated judge	Scales judgements, but brings its own biases