Generation parameters

At every step the model produces a probability distribution over the next token. Generation parameters do not change that distribution: they change how you draw from it. Precision, variety and coherence all follow from that choice.

The video is in Italian; this page is the written version in English.

Input: "The capital of France is"

  "Paris"     0.85
  "a"         0.03
  "located"   0.02
  "the"       0.01
  ...         thousands more, with tiny probabilities

Temperature

The main knob: it makes the distribution sharper or flatter.

                Temp 0.1     Temp 1.0     Temp 2.0
"Paris"          99.9%         85%          45%
"a"               0.05%         3%          12%
"located"         0.02%         2%          10%
others            0.03%        10%          33%

At low values the model almost always picks the most likely token: predictable, repeatable answers, ideal when precision matters. Around 0.5-0.7 sits the balance point used for most tasks. Above 1.2 choices get less obvious and more original, at the risk of losing the thread. Past 1.5 the text falls apart.

Same prompt, “describe a sunset in one sentence”, as the knob turns:

• 0.0 — “The sun turns red and orange as it sinks below the horizon.”
• 0.7 — “The sky catches fire in golden shades as the sun embraces the sea.”
• 1.5 — “Celestial flames danced nervously across an ocean of purple silk and forgotten thoughts.”
• 2.0 — “Slowly horse oblique sunset perhaps crystals cloud winding.” ← no longer meaningful

Top-p and top-k

Two ways of narrowing the field before drawing.

Top-k keeps only the k most likely tokens and ignores the rest: with k=40, the model chooses among the forty best candidates.

Top-p, also called nucleus sampling, is smarter: it keeps the most likely tokens until their combined probability reaches p. With p=0.9, if one token is already worth 85% the field stays tiny; if probabilities are spread out, the shortlist widens on its own. It adapts to the situation, which is why it is the common default today.

The penalties

Repetition penalty lowers the probability of tokens already used: the cure for a model that gets stuck repeating itself. Typical values 1.05-1.15; push it too far and the text turns stilted, because words that should repeat get punished too.

Presence and frequency penalties are variants of the same idea: the first punishes a token merely for having appeared, the second in proportion to how often it appeared.

Where to start

Task	Temperature	Top-p
Data extraction, classification	0 – 0.2	1.0
Coding	0.1 – 0.3	0.95
Technical answers, summaries	0.3 – 0.5	0.9
General conversation	0.6 – 0.8	0.9
Creative writing	0.9 – 1.2	0.95

One practical tip: move one knob at a time. Change temperature and top-p together and you will not know what caused the result.

When the format cannot be wrong

The parameters seen so far decide how much freedom the model has. There is a level above: forcing it to produce only text that respects a given structure. Not a suggestion in the prompt, a constraint applied during generation.

The mechanism is elegant: at each step the probability of tokens that would make the output invalid is zeroed, and the draw happens only among the allowed ones.

Generating JSON, having just written:  {"name":

Allowed:  "  (opens the value string)
Zeroed:   }  ,  free words, newlines...

→ the model CANNOT produce malformed JSON: not because it is good,
  but because the alternatives were removed

Use it whenever output feeds a program: it eliminates the most annoying error class in one move — the extra comma, the missing field, the comment added out of politeness — without retrying the call.

Two caveats. The constraint guarantees form, not content: valid JSON can hold a wrong answer. And overly rich structures hurt small models, because they spend attention fitting fields instead of reasoning: few fields, obvious names, no deep nesting.

Reading how confident the model is

Many interfaces can return, alongside the text, the probability of each chosen token. Information almost nobody uses, and worth a lot.

Answer: "The contract expires on 15 March 2027"

  "15"     0.94   ← confident
  "March"  0.91
  "2027"   0.38   ← guessing here

It serves three concrete purposes: flagging uncertain answers instead of presenting everything in the same tone, automatically filtering results below a threshold, and choosing among several generations.

The limit to know: high probability means “consistent with what I saw in the data”, not “true”. A well-rooted fabrication can arrive with very high probability. It measures statistical fluency — but low probabilities remain an excellent warning bell.

Why the same question gives different answers

A recurring question with three distinct causes worth separating.

Sampling: above zero temperature the choice is probabilistic, so two runs can legitimately diverge. Different context: in chat the history changes each turn, so you are not asking the same question, you are asking it inside a different conversation. Parallel computation: even at zero temperature tiny differences remain, because floating-point operations on GPUs are not associative and summation order depends on how work is distributed.

The last one explains why “deterministic” in practice means “almost always identical” rather than “identical by construction”.

In short

Parameter	What it does
Temperature	Flattens or sharpens the distribution: creativity versus precision
Top-k	Keeps the k best candidates
Top-p	Keeps candidates until probability p is covered: self-adapting
Repetition penalty	Discourages reusing tokens already seen
Constrained output	Structure guaranteed by the mechanism, not by goodwill
Token probabilities	They signal uncertainty, not truth
Reproducibility	Even at zero temperature a margin remains, and that is normal