Multimodality

The video is in Italian; this page is the written version in English.

A classic LLM is unimodal: text in, text out. A multimodal model handles several sensory channels at once.

UNIMODAL       text → [model] → text

MULTIMODAL     text  ─┐
               image ─┼→ [model] → text / image / audio
               audio ─┘

The difference is the one between a book and a person. The book holds only text; a person sees, hears, speaks, and integrates it all to work out what is going on.

How a model “sees”

The point is not that the model has eyes, but that images get translated into the same language as text: vectors.

A component called the vision encoder — usually a Vision Transformer — splits the image into patches, turns them into a sequence of embeddings, and that sequence is appended to the text one. From there the model works as always: predicting the next token, with both words and image patches in front of it.

This is where the embeddings lesson pays off: if different meanings can live in one vector space, so can different modalities.

The three kinds you will meet

Models that see. They take an image and produce text: descriptions, answers about the picture, chart reading, transcription of handwriting. They have become surprisingly good.

Models that generate images. They run the opposite way, from text to pixels, using different architectures (diffusion models), often placed alongside the LLM rather than inside it.

Models that listen and speak. Transcription, speech understanding, voice synthesis. Real-time conversational generation is the fastest-moving area right now.

What to realistically expect

Vision capabilities are excellent for describing a scene, reading a photographed document, explaining a chart. They are still fragile when spatial precision is required — counting objects, saying what exactly sits to the left of what, reading precise values off a dense chart.

There is also a surprising cost: an image eats a lot of tokens. Depending on resolution it can be worth several pages of text, so it consumes context quickly.

Locally, vision-capable models need more memory than their text-only counterparts, because the visual encoder sits on top of the language model.

What an image costs

The arithmetic that surprises newcomers: an image is not free, and not cheap. It is turned into a sequence of vectors occupying slots in the context window exactly like words.

Small image (512 × 512)      ~ hundreds of tokens
Large image (2048 × 2048)    ~ thousands of tokens
Scanned page                 often more than the text it contains

Three practical consequences: five high-resolution photos can saturate a window before you even ask the question; resizing before sending is almost always right, because past a certain resolution the model gains nothing; and on metered services image analysis costs far more than equivalent text.

Documents and tables: where it works and where it does not

The most requested use is reading documents. Worth knowing in advance where they are reliable.

Task	How it goes
Clean printed text	Very well
Summarising a page	Very well
Handwriting	Decent, depends on the hand
Complex tables	Fragile: skips rows, misaligns columns
Multi-column text	Fragile: scrambles reading order
Numbers read off a chart	Unreliable: it estimates, it does not read

The rule that follows: excellent for understanding a document; for extracting exact data — amounts, quantities, codes — verification is required.

And a tip that saves time and money: for native PDFs, vision is often not the best route. Extracting the text with a dedicated tool and passing that to the model is more accurate, cheaper and faster. Vision is for when there is no text: photos, scans, screenshots.

Locally: which models and how much memory

The practical question that follows the theory. A model that sees costs more than its text-only counterpart, because the visual component sits on top of the language model.

What you want	You need
Describing images, reading photographed documents	A vision model of 7-11B
Transcribing audio	A dedicated transcription model, very light
Generating images	A diffusion model: a separate path, not an LLM
Real-time speech	Still immature locally

Rule of thumb for memory: a vision model takes roughly 20-30% more than its text-only counterpart of the same size. On a 16 GB machine that means staying at 7-8 billion parameters and not going higher.

Audio transcription is the happiest case: dedicated models are small, fast even without a GPU, and quality is excellent across languages. It is probably the first multimodal use worth trying, because the ratio of result to effort is the best.

What not to expect yet

Worth being explicit, because public demonstrations always show the best case.

Vision models do not replace a data extraction tool: they read well, but on tables and exact numbers they need verification. Image generation locally is a separate world, with its own tools and requirements. And video is still out of practical reach: what people do today is analyse frames sampled at intervals, which is a different thing from understanding a video.

In short

Concept	In one line
Multimodal	One model for several data types
Vision encoder	Translates images into embeddings, the same language as text
Strength	Describing, reading documents, explaining figures
Weakness	Spatial precision: counting, positioning, exact values
Hidden cost	One image can be worth thousands of context tokens
Token cost	One image can be worth pages of text: resize first
Documents	Great for understanding, fragile for exact extraction
Locally	A vision model costs 20-30% more than text-only
The best case	Audio transcription: small models, fast, excellent output