Benchmarks and evaluation

The video is in Italian; this page is the written version in English.

Benchmarks are standardised tests: a set of questions with known answers, the model responds, you count the percentage correct. They work like school exams — they give a comparable number, and like school exams they do not tell the whole story.

The names that keep coming up

MMLU measures general knowledge across dozens of subjects with multiple-choice questions. An 8-billion-parameter model typically sits between 65 and 70 per cent, a 70-billion one between 80 and 85. It measures what the model memorised, not how well it understands.

HumanEval measures the ability to write correct code: one hundred and sixty-four programming problems, counting how many are solved on the first attempt.

GSM8K collects grade-school maths problems requiring several steps: the classic ground where explicit reasoning makes an enormous difference.

Then there are the arenas, where real people compare two anonymous answers and vote for the one they prefer. Closer to actual experience, but they also reward style and pleasantness, not only correctness.

Why to take them with a pinch of salt

Contamination. If the test questions ended up in the training data, the model is not reasoning: it is remembering. This happens more often than admitted, because famous benchmarks are public and spread everywhere.

Optimising for the test. When a number becomes a commercial target, people start training for that number. The score rises, general ability does not.

Distance from your case. No benchmark measures “answers my customers’ questions about my documents well”. Two models three MMLU points apart can behave in opposite ways on your task.

Language. Almost all benchmarks are in English: they say little about how a model performs in other languages.

The evaluation that counts

Building your own takes less time than it sounds, and beats any leaderboard.

Collect twenty real cases — not invented ones: prompts you would actually use, including the hard and the ambiguous. For each, write what you consider a good answer, even in one line.

Run the candidate models over the same twenty cases, with the same parameters, and read the answers side by side. Differences emerge that no aggregate score shows: who makes things up, who ignores the format, who loses the thread on long questions.

Keep all of it. When a new model appears, ten minutes tell you whether switching is worth it — and the same folder lets you check that a prompt change did not break something else.

Having a model judge the answers

Once your test cases number in the dozens, rereading them all by hand after every change is not sustainable. The common shortcut is using a model as judge: give it question, answer and criterion, and it returns a score.

It works, on two conditions. The criterion must be specific: “rate quality from 1 to 10” produces uninformative numbers, “does the answer cite only information present in the supplied text? yes or no” produces a verifiable judgement. And the judge must be calibrated: before trusting it, score twenty cases by hand and compare.

Bias	Effect
Length preference	Rewards verbose answers at equal content
Format preference	Rewards bullet lists and bold text
Confidence preference	Rewards assertive tone, even when wrong
Self-preference	Rewards answers from models of its own family

These biases are documented and systematic, therefore predictable: if the judge rewards verbosity and you optimise for that score, you get a verbose system. A practical antidote is judging by comparison — which of these two answers is better? — rather than absolute scoring, alternating the order to neutralise position effects.

Contamination

The most insidious flaw in public leaderboards: if the test questions ended up in the training data, the model is not reasoning, it is remembering.

This happens more often than admitted, because famous benchmarks are public and spread across the web — and training data is collected from the web. The typical symptom is a model that shines on known tests and disappoints on real tasks of the same kind.

It is also why new benchmarks, or ones that keep their questions private, are more informative than long-standing ones: they have not yet had time to leak into the data.

Building your own evaluation in half an hour

The thing that shifts the most and that almost nobody does. The concrete method.

Collect twenty real cases. Not invented: prompts you would genuinely use, taken from a week of work. Include the three kinds that matter — normal cases, hard ones, and ambiguous ones where a person would ask for clarification.

Write what counts as a good answer. One line per case is enough. Not the perfect answer, the criterion: “must cite the clause”, “three sentences maximum”, “if it is not in the text it must say so”.

Run the candidates on the same cases, with the same parameters, saving the answers.

Reread side by side and count real errors. Not “I like this one more”: how many times it made something up, ignored the format, dropped a constraint.

case                        model A          model B
contract summary            ok               ok
question on missing data    made it up       said it was missing   ← real difference
required JSON format        ok               extra comma
ambiguous question          guessed          asked

Half an hour, and you keep a reusable tool: when a new model appears you know in ten minutes whether it is worth it, and when you change a prompt you immediately see if you broke something else.

What NOT to measure

Three traps people fall into with their own evaluations, having just escaped the public benchmark ones.

How likeable the answer is. A brilliant tone is not quality: the same bias as automated judges, applied to you.

A single case. A model that fails once in twenty and one that fails once in three can give identical results on the one case you tried.

The task you already do well. If you only test what works, you measure how good you are at picking tests, not how good the model is. The cases that made you swear last week are worth ten comfortable examples.

In short

Concept	In one line
MMLU	General knowledge, multiple choice
HumanEval	Correct Python code on the first attempt
GSM8K	Multi-step maths problems
Contamination	Test data leaking into training skews everything
Your own eval	Twenty real cases beat any leaderboard
Automated judge	Scales evaluation, but carries known biases
Contamination	Tests leaked into training data: the score stops meaning anything
Your own evaluation	Twenty real cases, written criteria, errors counted side by side
Traps	Likeability, single cases, testing only what already works