Context window and memory

The context window is the maximum number of tokens a model can have in front of it at any moment. It is its desk: you can spread out as many sheets as fit, but when it is full and you want to add something, something else has to go.

The video is in Italian; this page is the written version in English.

┌─────────────────────────────────────────────┐
│        CONTEXT WINDOW (e.g. 8192 tokens)    │
│                                             │
│   system instructions      ~200 tokens      │
│   chat history            ~5000 tokens      │
│   latest question          ~100 tokens      │
│   room for the answer     ~2892 tokens      │
└─────────────────────────────────────────────┘

Inside the window: the model sees it.
Outside: it does not exist.

How big

Model	Window	Roughly
GPT-2 (2019)	1,024 tokens	a page and a half
Llama 2 (2023)	4,096 tokens	six pages
Qwen 3 8B (2025)	32,768 tokens	fifty pages
Llama 3.1 (2024)	131,072 tokens	two hundred pages
Claude 3.5 (2024)	200,000 tokens	three hundred pages
Gemini 1.5 Pro	1,000,000 tokens	fifteen hundred pages

As a rule of thumb in English, a token is worth about 0.75 words. So 4,000 tokens is around 3,000 words, and 32,000 tokens roughly fifty pages.

The model has no memory

This is the most common misunderstanding, and worth dismantling precisely: the model does not remember conversations. Every time you hit enter, the interface resends the entire conversation from the beginning.

YOU: "Hi, my name is Marco"
AI:  "Hi Marco, how can I help?"

YOU: "What is my name?"

What actually gets sent to the model, again, in full:
  [system]    You are a helpful assistant.
  [user]      Hi, my name is Marco
  [assistant] Hi Marco, how can I help?
  [user]      What is my name?

AI: "Your name is Marco."

The model remembered nothing: it reread. Every single time.

What follows

The conversation gets more expensive. Each exchange lengthens the text to resend: answers slow down and, on metered APIs, the bill grows.

When the window fills, something is lost. Interfaces drop the oldest messages or summarise them. That is where sudden amnesia halfway through a long chat comes from.

The middle is the weak spot. Models tend to use the beginning and the end of the context well, and to neglect what sits in between. If something is crucial, do not bury it in the middle of a long document.

Bigger context is not always better. Filling a 100,000-token window with loosely related material makes answers worse: little but well-chosen beats plenty.

What it advertises versus what it uses

A model promising 128,000 tokens does not use them all equally well. It is one of the widest gaps between spec sheet and real experience.

The maximum window is the limit beyond which input is rejected or truncated. The useful context is the length within which quality stays stable, and the two numbers rarely coincide. On small models the difference is stark: one advertising 32,000 tokens often starts slipping well before, forgetting constraints given at the start or confusing sources.

Measuring it takes five minutes. Take a long document, hide a specific sentence in three places — beginning, middle, end — and ask the model to report it back. Repeat with a longer document. The point where retrieval from the middle starts failing is your useful context with that model: a far more honest number than the advertised one.

Paying for the prefix once

Several providers heavily discount the portion of a prompt that repeats identically between calls, on one condition: it must sit at the start. The same holds locally, where runtimes reuse work already done on the prefix instead of redoing it.

Hence a composition rule that costs nothing and pays immediately:

[ FIXED     ]  instructions, documents, examples   ← at the top, reused
[ VARIABLE  ]  the question, today's data          ← at the bottom

Inverting the order — question on top, document below — invalidates the reuse on every request. A silent mistake: everything works, it just costs more and runs slower.

How lasting memory is simulated

The model does not remember, but an application can create the impression. Three techniques, with different costs.

Technique	How it works	Limit
Sliding window	Keep only the last messages	The past vanishes abruptly
Progressive summary	Periodically compress the history	Compression loses detail
External memory	Salient facts stored outside and retrieved	Someone must decide what to save

The third genuinely resembles memory: stable facts are extracted — “prefers short answers”, “is working on a particular project” — and kept in an archive, retrieved by similarity when the conversation makes them relevant.

The distinction worth holding onto: context is working memory, the archive is long-term memory. Confusing them fills the window with material that should have stayed outside.

In short

Concept	In one line
Context window	How many tokens the model sees at once
No memory	Every turn resends the whole chat from scratch
Growing cost	The longer the conversation, the slower and pricier
Lost in the middle	Beginning and end are recalled better than the middle
Quality > quantity	Relevant context beats abundant context
Useful context	Always shorter than advertised: measure it
Prompt order	Fixed parts first: they get reused and cost less
Real memory	Lives outside the conversation, not inside