Quantization

The video is in Italian; this page is the written version in English.

It is like lowering the resolution of a photo. In 4K it is perfect and enormous; at 1080p the difference is barely noticeable and the size halves; at 720p you start to see something but it stays fine; at 480p degradation shows, yet the file is tiny.

Quantization does this to the model’s weights: it reduces the precision each number is stored with.

The arithmetic is simple

Weights are billions of decimal numbers. In the original format each takes 32 bits, that is 4 bytes.

Format	Bits per weight	Representable values
Float32	32	about 4 billion
Float16	16	about 65,000
Int8	8	256
Int4	4	16

Hence the formula that actually matters: parameters × bits ÷ 8 = bytes. An 8-billion-parameter model takes roughly 32 GB in float32, 16 GB in float16, 8 GB at 8 bits and a little over 4 GB at 4 bits. That is the difference between “does not fit” and “runs nicely” on an ordinary machine.

How much you lose

Far less than halving the bits would suggest, which is why the technique caught on. Going from 16 to 8 bits the quality loss is generally negligible. At 4 bits there is a measurable but contained degradation, almost always acceptable. Below 4 bits the drop becomes obvious and the model starts making visible mistakes.

One principle beats every table: a large heavily quantized model almost always beats a small lightly quantized one at the same memory footprint. Between a 14B at 4 bits and an 8B at 8 bits — both around 8 GB — the first is usually the better pick.

The labels you will see

Downloading models you meet names like Q4_K_M, Q5_K_S, Q8_0. The Q and the number give the bits; the trailing letters indicate the variant and the quantization block size. In practice Q4_K_M is the most recommended balance point, and Q5_K_M the next step up when memory is plentiful.

The common file format for local use is GGUF: a single file holding weights and metadata, and the one used by the tools covered in the Local Setup section.

Worth knowing: not all weights are treated equally. Modern techniques keep the parts of the model that suffer most at higher precision, and that is what keeps the degradation so contained.

Compressing the cache, not just the weights

Everything so far concerns the weights. But a second item occupies memory during use and grows with the conversation: the key and value cache. On long contexts it can weigh as much as the model, sometimes more.

That can be quantized too.

8B model at 4 bits     ~4.5 GB    constant
Cache at 16 bits       ~2 GB      at 8k context, and growing
Cache at 8 bits        ~1 GB      same conversation, half the memory

The rule of thumb from practice: 8 bits on the cache is nearly free in quality terms and frees useful memory; dropping to 4 is noticeable, especially on long contexts where errors accumulate. It is the right lever when the model fits but long conversations slow everything down: no need to drop a size, just compress the cache.

How degradation is actually measured

“How much do I lose” is usually settled with a statistical index measuring how “surprised” the model is by a reference text. Convenient to compute and misleading on its own: between 8 and 4 bits that index barely moves, while on tasks requiring precision — following a rigid format, not dropping a constraint, writing code that compiles — the difference is plain to see.

Degradation is also not uniform: it hits fragile abilities first — arithmetic, multi-step reasoning, less represented languages — and generic conversation last. Which is why people who test a heavily compressed model by chatting conclude it is fine, then are surprised when it breaks the format.

The only measure that really counts remains the homemade one: the same twenty real prompts, before and after.

The labels: how to choose in practice

Downloading models you meet names like Q4_K_M, Q5_K_S, Q8_0 and the choice looks arbitrary. It is not, but the number of options is larger than needed.

Q4_K_M
│ │ │
│ │ └─ block size: S small, M medium, L large
│ └─── method family (K = block quantization, the common one)
└───── bits per weight

In practice you need three rows of this table, not all of them:

Label	When
Q4_K_M	Everyday use. The recommended balance point almost always
Q5_K_M	You have memory to spare and want a few quality points more
Q8_0	Precision matters and space is not an issue
Q3 and below	Only out of necessity: degradation shows

The M versus S variant matters far less than the jump between 4 and 5 bits: if choosing between Q4_K_S and Q4_K_M with enough memory, take the second and stop thinking about it.

And a reminder worth more than all the labels: at equal gigabytes, a large heavily compressed model almost always beats a small lightly compressed one. Between a 14B at 4 bits and an 8B at 8 bits — both around 8 GB — the first is usually the better pick.

GGUF and the other formats

The file format says which program can run it, and that is a different thing from quantization.

Format	What runs it
GGUF	llama.cpp and everything built on it: Ollama, LM Studio
MLX	Apple’s framework: faster on M-series chips
safetensors	Original unquantized weights: for training, not for running

If you download a model and it will not start, nine times out of ten it is the wrong format: safetensors instead of GGUF is the most common mistake for people arriving at Hugging Face without knowing what to look for.

In short

Concept	In one line
Quantization	Fewer bits per weight: less memory, slightly less precision
The arithmetic	Parameters × bits ÷ 8 = bytes used
Balance point	4 bits (Q4_K_M) for everyday use
Guiding principle	Large heavily quantized > small lightly quantized
GGUF	The typical file format for quantized local models
Compressed cache	8 bits on the cache is nearly free and frees memory
Measuring degradation	Statistical indices mislead: use your own real cases
The labels	Q4_K_M for everyday, Q5 if memory allows, Q8 when precision matters
Format	GGUF for Ollama and LM Studio, MLX on Apple Silicon: not the same as quantization