Hardware for local LLMs

The video is in Italian; this page is the written version in English.

There is only one constraint that really counts: the model must fit in memory. Which memory depends on the machine, and that changes the rules entirely.

On an NVIDIA card the model has to fit in the card’s VRAM, which is separate and typically smaller than system RAM. Apple Silicon has unified memory, shared between CPU and GPU: a 16 GB Mac can devote a good share of it to the model, which is why Apple laptops do well here. Without a GPU you can use system RAM, but the slowness makes it unpleasant.

The useful analogy: video memory is the desk you work on, RAM is the cabinet next to it, the SSD is the warehouse in the basement. The bigger the desk, the more sheets you keep open at once.

The arithmetic

Same as quantization, with headroom for the rest:

memory needed ≈ (parameters × bits) / 8   +   overhead

Overhead covers the context — which grows with conversation length — plus working structures. Rule of thumb: add roughly twenty per cent to the calculated result.

Available memory	What runs well
8 GB	7-8B models at 4 bits
16 GB	8B comfortably, 14B at 4 bits
24-32 GB	32B at 4 bits
48-64 GB	70B at 4 bits

What determines speed

Surprise: not raw compute so much as memory bandwidth. Generating a token requires traversing all the model’s weights, so how fast memory can be read dominates. That is why cards with similar compute but different bandwidth give very different results.

Two timings are worth separating. The first token arrives after the model has processed the whole prompt: with long context, the wait is noticeable. Subsequent tokens come at a steady rate, and that is the number usually advertised.

For scale: an 8B model quantized to 4 bits on recent hardware typically runs between twenty and fifty tokens per second, faster than you read.

The common mistakes

Filling memory to the brim. If the model takes everything, the system starts swapping to disk and speed collapses by an order of magnitude. Better to drop a size and stay fluid.

Looking only at parameter counts. An MoE model with many total parameters activates few per token: it weighs like the total in memory but runs like the active fraction.

Forgetting the context. A long conversation or a pasted document occupies memory on top of the model. People working at the limit find out halfway through a chat.

Two timings: initial wait and writing speed

Tokens per second tell only half the story. There are two distinct timings, with different bottlenecks, and confusing them leads to wrong diagnoses.

Phase	What happens	What it depends on
Initial wait	The model processes the whole prompt	Compute power, prompt length
Writing	Tokens come out one after another	Memory bandwidth

The first grows with prompt length: an 8,000-token context makes you wait before the first word, even on fast hardware. The second is nearly constant.

Two practical conclusions. If the initial wait is unbearable, the problem is context: shorten the prompt, or keep fixed parts at the top to exploit reuse. If writing is slow, context is irrelevant: you need a smaller model, more compression, or hardware with more bandwidth.

Diagnosis in three steps

The symptom “it got slow” almost always has one of three causes, distinguishable in minutes.

slow IMMEDIATELY, always      → memory: the model does not fit comfortably
slow ONLY before answering    → context too long
fast at first, then declining → heat: the machine is protecting itself

In the first case the system is swapping to disk: close something, drop a size, or reduce context. In the second it is the prompt being processed: shorten or reorganise. In the third clock speeds fall after minutes of sustained work, and there is little to do beyond better ventilation and alternating workloads.

A detail people miss: context occupies memory too, and it grows during the conversation. A model that fitted comfortably at the start may not fit after half an hour of chat.

Three setups and what you get from them

Abstract numbers say little. Here is what actually changes moving from one machine to another.

8 GB laptop. A 7-8 billion model at 4 bits runs, and that is it: around 5 GB for the model, the rest for system and context. Fine for rewriting, translating, classifying, answering over text you supply. Long conversations are the limit: as the cache grows, swapping starts.

16 GB laptop or desktop. This is where local becomes pleasant. An 8B at 4 bits runs comfortably with a wide context, a 14B at 4 bits is within reach for harder tasks, and there is headroom to keep other things open. It covers the large majority of personal use.

32 GB machine or more. 32-billion models at 4 bits open up, and they make a perceptible jump on complex tasks. Above that, with 64 GB, you reach 70B — but the ratio of gain to cost worsens quickly, and for those few tasks an API is often the better answer.

Memory	Typical model	What you get
8 GB	7-8B at 4 bits	Bounded tasks, short context
16 GB	8B comfortably, 14B at the limit	Full everyday use
32 GB	32B at 4 bits	Complex tasks, reasoning
64 GB+	70B at 4 bits	Diminishing returns: consider an API

Dedicated card or unified memory

Two worlds with different rules, and the difference is not raw power.

On Apple Silicon memory is shared between processor and graphics: a 16 GB Mac can devote a good share to the model, which is why laptops without a dedicated card do well. The limit is bandwidth, which grows with the tier.

On PCs with an NVIDIA card only the card’s memory counts. Having 64 GB of system RAM does not help if the card has 12: whatever does not fit ends up on the processor, and speed collapses. In exchange, when the model fits entirely, bandwidth is far higher.

The practical consequence when buying: on a Mac look at total unified memory, on a PC look only at the graphics card’s memory.

In short

Concept	In one line
Main constraint	The model must fit in memory, full stop
Unified memory	On Apple Silicon CPU and GPU share it
The arithmetic	Parameters × bits ÷ 8, plus about 20% headroom
Speed	Memory bandwidth matters more than raw compute
Typical mistake	Filling everything: you end up swapping to disk
Two timings	Initial wait and writing speed have different causes
Diagnosis	Always slow, slow only at the start, or slowing over time: three problems
Setups	8 GB bounded tasks, 16 GB full use, 32 GB complex tasks
What to check	On Mac unified memory, on PC only the card’s memory