How to choose a model

The video is in Italian; this page is the written version in English.

“Which model is best” has no answer. The right question is “which model is best for this task, on this machine”, and you get there in four sequential steps.

1. TASK          what do you need it for?
2. SIZE          how large can it be?
3. QUANTIZATION  how much do you compress it?
4. RUNTIME       what do you run it with?

Step 1 — The task

Specialised models beat generalists on their own ground, at equal size.

For code there are variants trained on enormous amounts of source: they grasp structure and convention far better than a generic model of the same weight. For conversation and writing, generalists are fine. For multilingual work what matters is the training data mix. For reasoning you want models with explicit thinking, which cost more time per answer but go wrong far less on multi-step problems.

Step 2 — The size

Available memory rules here, and the arithmetic is the quantization one: parameters times bits divided by eight.

Memory	What runs comfortably
8 GB	7-8B models at 4 bits
16 GB	8B with headroom, 14B at 4 bits
32 GB	32B at 4 bits
64 GB+	70B at 4 bits

The earlier principle still holds: at equal memory, a large heavily quantized model beats a small lightly quantized one.

Step 3 — The quantization

In practice: Q4_K_M for daily use, Q5 or Q6 if memory is spare and you want a few quality points more, Q8 only when precision really matters and space is no issue. Below 4 bits only out of necessity.

Step 4 — The runtime

What you run it with depends on what you are doing. Ollama for the command line and a local API to build on. LM Studio if you prefer a graphical interface to download, try and compare. Open WebUI when you need a real chat, with multiple users or reachable from your phone. They are the subject of the Local Setup section.

A mistake to avoid

Picking the largest model that fits in memory is almost never right: if it fills everything, the system starts swapping to disk and the conversation becomes painful. Better to leave headroom and get fluid answers.

Above all: try it on your real case. Ten minutes with your typical prompts tell you more than any leaderboard.

When it is worth switching

A new one appears every week and the urge to upgrade is constant. But switching has an invisible cost: prompts tuned for the old one may perform worse, output formats shift in tone, and everything you had verified needs verifying again.

The rule: switch when the new one wins on your tests, not on benchmarks.

The minimal protocol, a quarter of an hour: download the candidate alongside the one you use without removing anything; run the same twenty real prompts with the same parameters; read the answers side by side counting real errors, not impressions; check speed and memory too; if it wins, keep it on trial for a week before discarding the old one.

Signal	Switch?
Better public benchmark scores	On its own, no
Wins on your twenty cases	Yes
Same quality but smaller or faster	Yes, a clear gain
A new capability you actually need	Yes
“It came out yesterday and everyone is talking about it”	No

The case always worth taking is the third: the same output in less memory frees room for longer context or for running something else on the same machine.

The mistake to avoid

Choosing the largest model that fits in memory is almost never right. If it takes everything, the system starts swapping to disk and conversation becomes painful; and every extra token of context makes it worse, because the cache takes space too.

Better to drop a size and stay fluid: the quality difference between a 14B and an 8B shows on hard tasks, the difference between an immediate answer and a trickling one shows always.

Three concrete scenarios

The four steps in the abstract are easy to forget. Here is how they actually apply.

Writing code on a 16 GB laptop. Task: coding, so a specialised model beats a generalist of the same size. Size: 7-8 billion with headroom, or 14 at the limit. Quantization: Q4_K_M. Runtime: Ollama if you wire it to your editor, LM Studio to try it by hand first. Result: a 7-8B code model at 4 bits, around 5 GB.

Answering over confidential company documents. The deciding constraint is privacy: local, no API. The model matters less than you think, because the real work is document retrieval — an 8B generalist is enough, and the effort goes into chunking and hybrid search. If retrieval does not find the right chunk, no model saves the answer.

You need top quality on a few hard tasks a day. Here local is not the answer: low volume, high difficulty, no sensitive data. API, frontier model, and the bill stays small precisely because the requests are few.

The mistakes that keep repeating

Mistake	Why it costs
Taking the largest model that fits in memory	The system swaps to disk, everything slows
Choosing on benchmarks instead of your own case	Models close in score can behave in opposite ways
Using a generalist for code	A specialised model of the same size does better
Switching model at every release	Prompts are tuned: each switch needs reverifying
Ignoring language	Many models perform worse outside English

The last is the most underrated: two models equivalent on English benchmarks can differ sharply in another language, and you only find out by trying.

In short

Step	Question	Rule of thumb
Task	What for?	Specialised beats generalist on its own ground
Size	How much memory?	Leave headroom, do not fill to the brim
Quantization	How much compression?	Q4_K_M as a starting point
Runtime	How do you run it?	Ollama, LM Studio or Open WebUI by use case
When to switch	Only if it wins on your real cases, not on benchmarks
Common mistake	Filling memory to the brim: better one size down
Scenarios	The deciding constraint changes: hardware, privacy, or task difficulty
Recurring mistake	Filling memory, and choosing on benchmarks instead of your own case