A specialised RAG, running locally: the knowledge bot built on twelve books

The video is in Italian; this page is the written version in English.

The previous lesson built the pipeline piece by piece. This one watches it work on a real case: an assistant that answers only from a dozen game design books, with citations, without sending anything off the machine. The point is not game design: the same pattern applies to any closed corpus, from a firm’s contracts to a company’s manuals.

The problem: a model that guesses

Ask an AI assistant a precise question. You know the answer is in a book on your shelf. The model answers anyway, confidently, and the answer is plausible and wrong. It is not bad faith: the model has never read that book, and when a piece is missing it fills the gap with the most probable thing. That is a hallucination, and it is why “knows a bit of everything” and “knows what it is talking about” are two different things.

The fix is to change the rules of the exam: from reciting by heart to an open-book exam. First the model is handed the right pages, then it is asked to answer using only those.

by heart                              open book (RAG)
question → model → answer             question → search the documents → passages
                                                → model + passages → cited answer

Two specialists instead of one generalist

The architectural idea that keeps the system light is not asking one model to do everything. There are two roles, and two different models fill them.

Role	Model	What it does	When it works
The librarian	nomic-embed-text (274 MB)	Turns text into vectors and finds the passages closest to the question	Always: it is the search engine
The synthesiser	qwen3, 8 or 14 billion parameters	Reads the retrieved passages and writes a prose answer, with sources	Only when a generated answer is requested

The librarian is small, fast and does one thing. The synthesiser is larger and slower, but it only steps in at the end and receives an already-selected context. Separating the two roles has a practical consequence: search is instant even without generation, and it can be used on its own whenever finding the passage is enough.

The full flow, from books to answer:

12 books (PDF/EPUB)
   → text extraction         232 chapters as plain text
   → chunking + embedding    5,234 pieces of ~800 tokens, vectorised
   → vector store            ChromaDB, cosine distance
   → question                the librarian retrieves the 5 closest pieces
   → (optional) generation   the synthesiser answers, citing

Every piece carries its metadata: book, author, chapter, pages. That is what lets the system say where a sentence comes from, and lets the reader go and check.

Three ways to query it

A powerful tool that is awkward to use does not get used. The system exposes three interfaces over the same index, for three different habits.

Web interface. A page in the browser with two buttons: Search returns the most relevant passages instantly; Generate adds the answer written by the synthesiser. From the menu you pick the model, how many passages to retrieve and the prompt mode. It is the recommended route for anyone who does not want a terminal.

Command line. A chat in the terminal, with colours and formatting, exposing every option: model, number of results, mode, single non-interactive question.

# search only, instant
python3 query.py -q "What are feedback loops?"

# search + generated answer, with structured analysis
python3 query.py --mode mechanics --top 10 --model qwen3:14b

No installation. The extracted chapters are text files with a metadata header. An AI-assisted code editor, such as Claude Code, can search through them and read the passages with no vector index and no local models: an on-the-fly text search, which does not replace RAG but covers “where do they talk about this” questions well.

Four expert modes: change the prompt, not the documents

This is the most interesting part. The same tool offers four question modes, and the difference between them is not in the retrieved documents, which are always the same: it is in the instruction given to the synthesiser.

Mode	When	What it imposes on the model
general	A classic question	Answer only from the context, cite sources
mechanics	Analysing a mechanic	Identify → compare → trade-offs → suggest variants
codesign	Designing your own game	Propose concrete mechanics, explain why they work, cite the theory
compare	Contrasting two approaches	Overview → similarities → differences → context → verdict

The system prompt of the mechanics mode says, in essence:

1. IDENTIFY   which sources in the context discuss this mechanic
2. COMPARE    how each one describes it
3. TRADE-OFFS pros and cons of each approach
4. SUGGEST    two or three concrete variations to try

That is not a question, it is a reasoning template. The answer comes out structured and cited instead of as a wall of text, and the cost of adding a new mode is writing one paragraph of instructions, not re-indexing anything. It is the same lever seen in system prompt design, applied to an already filtered context.

Purpose-built or generic?

Ready-made tools exist that do RAG on documents uploaded on the fly, such as Open WebUI. Why build a specific one?

	Purpose-built tool	Generic tool
Purpose	Search those twelve books	General chat, RAG on demand
Index	Ready, 5,234 pieces	Documents are uploaded each time
Requirements	Only the local model runtime	Runtime + container, more memory
Expert modes	Yes, in the prompt	Recreated by hand every time
Chat history, images	No	Yes

The rule of thumb is simple: for a specific field, queried often, on the same corpus, the purpose-built tool is lighter and more convenient, because everything is already indexed and it knows how to answer. For everything else, the generic one. They do not compete: they cover different questions.

Growing the corpus without redoing everything

A vector index is not a single block: every piece carries the identifier of the book it comes from, and that allows working in parts. Adding a thirteenth book is a short sequence.

1. put the file in the sources folder
2. register it in the list of books (title, author, identifier)
3. extract the text of that book only
4. vectorise that book only and add it to the store

Indexing all twelve books takes a quarter of an hour on a laptop; a single book, a minute. The same logic works in reverse: a book to remove or redo is deleted by identifier, without touching the others. That is why the metadata should be decided before indexing: a piece with no book identifier can only be thrown away together with everything else.

Limits, and the typical mistakes

• The librarian decides the quality. If the retrieved passages are wrong, the synthesiser writes a well-formed answer on the wrong context. When an answer is off, look first at what was retrieved and only then at the model.
• The language of the index matters. With books in English, questions in English retrieve better: the embedding compares texts, and closeness across languages is weaker.
• How many pieces to pass. Five is a good starting point; ten are needed for comparative analyses and call for a model with a longer context. Fifteen distract.
• Generation is slow. With an 8-billion model an answer takes tens of seconds on a laptop; with the 14-billion one, more. Search alone stays instant, and is often enough.
• A corrupt input file disappears silently. An EPUB that is actually an HTML page yields zero chapters and no error: after extraction, count what came out.
• It is not a do-everything tool. On questions outside the corpus RAG does not help, and irrelevant context makes the answer worse.

In short

Concept	In one line
The problem	The model has not read your documents and fills the gaps
The fix	Open-book exam: the right passages first, then the answer
Two models	A small librarian to search, a synthesiser to write
The size	12 books, 232 chapters, 5,234 indexed pieces, all offline
Three interfaces	Web, command line, plain text search with nothing installed
The modes	They change the system prompt, not the documents
Purpose-built vs generic	Fixed corpus and frequent questions: purpose-built. The rest: generic
When it is wrong	Check retrieval first, then the model