Vector databases: searching by meaning

A regular database answers exact questions: give me the rows where the name is “Marco”. A vector database answers a different one: give me the texts that talk about something similar to this. It is the part that holds retrieval up, and the only component of the pipeline that behaves like real infrastructure — with indexes, trade-offs and numbers to work out.

SQL
  SELECT * FROM documents WHERE title LIKE '%polymorphism%'
  → finds it only if that word appears in the title

Vector
  search(vector("how does inheritance work in Java"))
  → also finds override, abstract classes, polymorphism:
    close meanings, different words

What it actually holds

A row in a vector store is not just a vector. It is three things, and you need all of them:

id:       "java-handbook#chunk-23"
vector:   [0.23, 0.87, -0.12, 0.45, ...]      ← to search
text:     "Polymorphism lets objects of..."    ← to answer
metadata: { source: "handbook.pdf", page: 145, section: "5.3" }

The vector is there to find, the text is what the model answers from, the metadata is what lets you filter and cite. A store that keeps only vectors forces you to hold the texts somewhere else and keep the two aligned by hand: it looks like a detail until the first re-index.

How closeness is measured

Three metrics, but the choice is simpler than it looks.

Cosine similarity. Looks at the angle between two vectors and ignores their length. It runs from -1 to 1, where 1 means “same direction”, which means same meaning. It is the standard for text and what you want in almost every case.

Euclidean distance. The straight-line distance between two points. It also accounts for vector length, which for text usually means nothing.

Dot product. Combines direction and length. On normalised vectors it coincides with cosine, which is exactly why several stores use it under the hood.

Rule of thumb: cosine, then move on. The one thing to watch is consistency: index with one metric and query with another and the scores mean nothing.

Indexes: why you don’t compare every vector

With a thousand chunks you can compare the question against all of them and take the closest. It is exact and costs nothing. With a million chunks the same approach falls over, so you use an index that agrees to be almost exact in exchange for speed.

Index	How it works	The trade-off
Exact (flat)	Compares everything	100% precise, slow at scale
HNSW	Navigable layered graph	Very fast, eats RAM
IVF	Groups vectors into clusters	Good balance, needs tuning
Quantisation	Compresses the vectors	Little RAM, some precision lost

What matters: below a hundred thousand chunks, the exact index is perfectly fine. Before that threshold, tuning an HNSW is time spent optimising something that is not the problem. At that point the problem is still chunking.

There is a consequence that surprises people: the moment you use an approximate index, retrieval can miss a relevant chunk that the exact one would have found. That is not a bug, it is the price you chose. But when comparing the quality of two setups, remember you are comparing this too.

The options, in the order you need them

	Chroma	FAISS	LanceDB	Qdrant	Pinecone
How it runs	In your process	Library	In your process	Server	Cloud service
Local	Yes	Yes	Yes	Yes	No
Metadata filters	Yes	No	Yes	Advanced	Yes
Persists on its own	Yes	No, by hand	Yes	Yes	Managed
To start with	Recommended	No	Yes	Later	No
In production	Small scale	With scaffolding	Medium scale	Yes	Yes

Chroma to start: install and go, saves to disk, keeps text and metadata together. FAISS is the fastest library around but it is not a database: no persistence, no filters — the surroundings are yours to build. Qdrant is the next step when you need real filters or several users. Pinecone removes the maintenance and in exchange moves your documents onto somebody else’s service — which for personal notes or company material is exactly the question to ask up front, not later.

Metadata filters matter more than raw speed

Purely semantic search has a practical flaw: it cannot say no. Ask about chapter 5 and, if chapter 5 says nothing about it, you still get the least distant chunks, pulled from anywhere in the store.

Filters fix that, and they are why metadata has to be saved from the start:

search(vector("expense reimbursement"), filter={ source: "policy-2026.pdf" })
search(vector("polymorphism"),          filter={ chapter: 5 })
search(vector("holidays"),              filter={ lang: "en", year: { ">=": 2025 } })

A store without filters forces you to narrow down after retrieving, which is to say after the retrieval was already wasted. It is the main reason people switch tools mid-project.

Where it goes wrong

Changing the embedding model without rebuilding the store. Old vectors live in one space, new ones in another: the distances become meaningless numbers. There is no error, just retrieval getting worse for no visible reason. Change the model and you rebuild everything.

Re-indexing without emptying. Running indexing again over the same documents without clearing produces duplicates: the same text takes two or three of the top slots, and the model receives three copies of one thing instead of three different chunks.

Storing vectors but not texts. You retrieve identifiers and then have to fetch the texts from somewhere else, hoping nobody renumbered the chunks in the meantime.

Optimising the index before everything else. On small stores the index choice is not noticeable: chunking and the embedding model are. That is the order worth working in, not the reverse.

Reading the score as a probability. A cosine of 0.82 does not mean “82% correct”. Scores are comparable within one search, not in the absolute: the threshold below which you discard a result has to be measured on your own documents, not copied from a tutorial.

In short

Concept	In one line
Vector database	Stores vectors and finds the ones closest to a question
What to store	Vector to search, text to answer, metadata to filter
Metric	Cosine, save for special cases
Exact index	Fine below a few hundred thousand chunks
Approximate indexes	Speed in exchange for the odd relevant chunk missed
Where to start	Chroma locally; Qdrant when filters or scale arrive
Metadata filters	Often more useful than raw speed
Recurring mistake	Changing the embedding model without rebuilding the store