Tokenization

An LLM does not see letters, and not even words. It sees tokens: variable-length chunks of text turned into integers. Every model has its own vocabulary — the list of all possible tokens — and its own tokenizer, the algorithm that cuts text up.

The video is in Italian; this page is the written version in English.

What it looks like

Text:    "The cat sleeps on the couch"
Tokens:  ["The", " cat", " sleeps", " on", " the", " couch"]
Numbers: [2460, 78432, 35271, 9182, 1847, 5523]

Notice the space glued to the front of each word: it is part of the token. Rarer words get split into pieces that do exist in the vocabulary.

Why not just use words

It sounds more natural, but it fails. The vocabulary would explode: millions of words across languages, each entry costing memory. New words would be unmanageable: coinages, proper nouns, typos. And you would lose morphology: “run”, “running”, “runner” share a root that sub-word pieces let the model exploit.

So why not single characters? Because sequences would become enormous — and with attention costing the square of the length, the bill climbs fast. Each character also carries very little information, forcing the model to work harder to rebuild meaning.

Sub-word pieces are the compromise: vocabularies of 32,000 to 150,000 entries, manageable sequences, and no text the tokenizer cannot digest.

How the vocabulary is built: BPE

The most common algorithm is Byte Pair Encoding, and its logic is three moves repeated many times.

Corpus:             "abab cdab abcd"
Starting vocabulary: {a, b, c, d, space}

1. count the pairs:  (a,b) → 4 times   ← most frequent
                     (c,d) → 2 times
2. merge the pair:   a + b → "ab"
3. start again:      {a, b, c, d, space, ab}

Repeat thousands of times over a huge corpus and frequent sequences become single tokens. That is why common words are one token while rare ones get split.

What follows in practice

Some languages cost more than others. Vocabularies are trained mostly on English text, so other languages get split more often. In English a token is worth about 0.75 words; in Italian roughly 0.6-0.7. The same sentence, translated, eats more context and — on paid APIs — more money.

Numbers are fragile. A long number may be split in unpredictable ways, one of the reasons arithmetic is not a strong suit.

Counting letters is hard. “How many r’s in strawberry?” trips models up precisely because they do not see letters: they see pre-packaged chunks.

How messages become tokens

This is the piece missing for anyone moving from the chat window to programmatic use and wondering why the model behaves oddly.

A model does not receive “a user message”: it receives a flat sequence of tokens. The role-based conversation is flattened using special markers, following a scheme called a chat template which is model specific.

What you write:
  system:  You are a concise assistant.
  user:    Hello

What the model actually receives:
  <|im_start|>system
  You are a concise assistant.<|im_end|>
  <|im_start|>user
  Hello<|im_end|>
  <|im_start|>assistant

That last line, left open, is the signal that it is now the model’s turn. Different formats use different markers: each family has its own.

Why it matters: with the wrong scheme the model answers badly without raising any error. It continues the text instead of replying, ignores system instructions, does not know when to stop. Tools like Ollama and LM Studio apply the right scheme automatically; with low-level libraries it becomes your responsibility.

There is a security angle too: those markers are reserved tokens, and a user who types them in their own message can try to confuse the roles. It is one of the routes of so-called prompt injection.

Rule of thumb: if a local model answers strangely, check the message scheme before blaming the model.

No text is out of vocabulary

It is natural to wonder what happens with a character never seen before: a rare ideogram, a mathematical symbol, a new emoji.

The answer is that the vocabulary starts not from characters but from the 256 possible bytes. Any text, in any encoding, decomposes into bytes — so no input exists that the tokenizer cannot represent.

Character absent from the vocabulary
    → decomposed into its bytes
    → every byte is a valid token
    → no error, but many tokens for a single character

The price is efficiency: a rare character can cost three or four tokens instead of one. That is why texts full of emoji, formulas or non-Latin alphabets burn through context quickly.

Worth knowing too that BPE is not the only algorithm around. Others exist, with modest practical differences except one useful case: some do not assume words are separated by spaces, which makes them suitable for languages that do not use them, such as Japanese or Chinese.

From tokens to the bill

This is the calculation worth being able to do in your head, because it decides between a paid service and a model at home.

APIs charge per token, with different prices for input and output: generating typically costs three to five times more than reading.

Example: a document assistant, 200 requests a day

  average context per request   4,000 input tokens
  average answer                  500 output tokens

  per day   200 × 4,500  =    900,000 tokens
  per month              = 27 million tokens

At that volume the choice of model is measured in hundreds a month, and becomes the line item that decides the architecture of the whole system.

Three ways to bring the bill down, in order of effectiveness. Cut useless context: sending a whole document when two paragraphs would do is the most common and most expensive mistake, and good similarity retrieval cuts input by an order of magnitude. Use the right model: classifying or extracting data does not need the most capable model, and often a small local one does the same job for free. Exploit prompt caching: several providers heavily discount the portion that repeats identically, provided it sits at the start.

And the language point returns here: the same request in a non-English language costs roughly twenty to thirty per cent more, because it splits into more tokens. At high volume, writing system instructions in English is a legitimate optimisation.

In short

Concept	In one line
Token	A variable-length chunk of text turned into a number
Vocabulary	The list of possible tokens, 32k to 150k entries
BPE	Repeatedly merges the most frequent pairs to build the vocabulary
Cost per language	Non-English text consumes more tokens for the same content
Known limits	Arithmetic and letter counting suffer from how tokenization works
Chat template	Roles become markers: the wrong scheme breaks things silently
Byte-level	No text is out of vocabulary, but rare characters cost many tokens
Cost	Billed per token: cutting useless context is the strongest lever