Prompting small models

The video is in Italian; this page is the written version in English.

People arriving at local models after using frontier ones almost always make the same mistake: they reuse the same prompts and conclude the small model “doesn’t work”. The problem is that an 8-billion-parameter model is not a shrunken 400-billion one: its limits are different in kind.

Capability	Large model	Small model (8B)
Complex instructions	excellent	limited
Long context	uses 128k well	in practice far less than advertised
Multi-step reasoning	good	loses the thread after three or four steps
Holding a format	reliable	tends to forget it along the way
Ambiguity	asks for clarification	guesses

The underlying reason: a large model has enough parameters to encode both knowledge and behavioural rules. A small one has to choose. Load it with instructions and it spends attention following them, leaving less for the content.

Hence the rule of thumb: with a small model, every word in the prompt must earn its place.

The rules that make the difference

Short, direct sentences. Polite, elaborate constructions are noise.

Bad:   I would like you to analyse the following Python code and provide me with a
       detailed list of all the potential issues you might identify...

Good:  Find bugs in this Python code. Focus: errors and security.
       List: line, problem, fix.

One task at a time. If you need four things, make four calls. Faster than arguing with an answer that does two well and two badly.

Show the format, do not describe it. One output example is worth three paragraphs of instructions, and with small models the gap is even wider.

Repeat important instructions at the end. Small models weight the tail of the prompt heavily: one reminder line just before the answer recovers a lot of format errors.

Few rules in the system prompt. Three or four, not twenty. What is detailed guidance on a large model becomes a load that smothers the content here.

Realistic expectations

Small models are excellent at bounded tasks: rewriting, classifying, extracting, translating, summarising short texts, answering over context you supply. They struggle with long reasoning chains, many simultaneous constraints, and very specific knowledge.

The strategy that pays is splitting: where a large model handles a complex task in one call, with a small one you make four simple ones. The end result is often comparable, and it runs on a machine sitting in your house.

The structural fix for the format problem

Most of the advice above aims at getting the right format: show the example, repeat at the end, keep it short. All of these persuade the model.

There is one that does not persuade: it compels. Constrained generation zeroes, at every step, the probability of tokens that would break the required structure.

Unconstrained:  ask for JSON, get JSON  ~ 8 times out of 10
                (the other 2: extra commas, preamble text, missing fields)

Constrained:    valid JSON  10 times out of 10
                because the alternatives are unavailable

It is especially valuable on small models, where format compliance is weakness number one: an 8-billion model with constrained output is more reliable, on format, than a much larger one left free.

Two caveats that count double here: the constraint guarantees form, not content; and overly rich schemas hurt, because the model spends attention fitting fields instead of reasoning. Few fields, obvious names, no nesting.

Realistic expectations

Worth being explicit about what to ask and what not to.

Works well	Struggles
Rewriting, translating, summarising short texts	Long multi-constraint reasoning
Classifying and extracting data	Highly specific knowledge
Answering over context you supply	Holding the thread across long documents
Well-defined repetitive tasks	Ambiguity that should prompt a question

The strategy that pays is splitting: where a large model handles a complex task in one call, with a small one you make four simple ones. The end result is often comparable, and it runs on a machine in your house.

Before and after: the same task rewritten

The difference shows best on a concrete case. Task: extracting data from a review.

# Written for a large model — on an 8B it gives inconsistent results

I would like you to carefully analyse the following product review and provide me
with a structured analysis of the sentiment expressed, also identifying the specific
aspects mentioned by the user both positively and negatively, taking into account
nuance and any sarcasm, and returning all of it in a format that is easily
processable by a program.

# Rewritten for a small model

Extract the data from the review between the markers.

Reply ONLY with this JSON:
{"sentiment": "positive|negative|mixed", "pros": [], "cons": []}

<<<REVIEW
...
REVIEW>>>

Reply only with the JSON.

What changed, point by point: one request instead of four merged; the format shown rather than described; the allowed values listed, so it does not invent them; the data delimited, so it does not confuse it with instructions; the key instruction repeated at the end, where the model weights more. And nuance and sarcasm are gone: you ask a small model something it can do, not the thing you wish it could.

If the JSON must always be valid, the final step is constrained output: at that point the form no longer depends on the model’s goodwill.

When the model is wrong, not the prompt

There is a limit beyond which rewriting the prompt does nothing, and recognising it saves hours.

Symptom	Interpretation
Gets the format wrong occasionally	Prompt can improve, or use constrained output
Gets the format wrong almost always	Model too small for that format
Loses the thread after three steps	Structural limit: split into several calls
Invents data that is not there	Context is missing, not prompt quality
Fine in English, poor in another language	The model is weak in that language: change it

The last row happens more often than people think, and stays invisible until you try the same prompt in both languages.

In short

Rule	Why
Short sentences	Every extra word steals attention from the content
One task per call	Multiple tasks degrade each other
Show the format	One example beats three paragraphs of description
Repeat at the end	The tail of the prompt carries more weight
Few system rules	Three or four, not twenty
Constrained output	On small models it is not an optimisation, it is the default
Strategy	Four simple calls beat one complex call
Rewriting	One request, format shown, allowed values, delimited data, final repetition
When to change model	Format always wrong, or much worse output in your language