Embeddings and Vector Search: The Retrieval Substrate

The substrate under retrieval

Every semantic search box and every retrieval-augmented generation system rests on the same layer, and it is the layer teams pay the least attention to. Before a language model ever reasons over a retrieved passage, something has to decide which passages are even candidates. That decision is made in a numerical space of embeddings, and the quality of that space is the quality ceiling for everything above it. A brilliant generator handed the wrong three chunks produces a confident, well-written wrong answer.

The reason this layer gets skipped is that it works well enough in a demo. You wire up an off-the-shelf embedding model, index a few hundred documents, ask the five questions you already know the answers to, and it looks like magic. The gap opens later, at scale, on the long tail of real user questions where the vocabulary does not match, the domain is specialized, or the answer hinges on an exact identifier. By then the embedding choice is baked into an indexed corpus that costs real money and time to rebuild.

This deep-dive treats embeddings as an engineering substrate rather than a black box. It covers what an embedding actually represents, why the model that produces it dominates retrieval quality, how approximate nearest-neighbor search turns vectors into fast lookups, and the structural limits that make hybrid search and reranking non-optional in production. It sits directly downstream of chunking and directly upstream of hybrid search.

What an embedding is

An embedding is a dense vector: a fixed-length list of numbers, often a few hundred to a few thousand of them, that represents a piece of text as a single point in a high-dimensional space. The property that makes it useful is geometric. An embedding model is trained so that text with similar meaning lands at nearby points and text with unrelated meaning lands far apart. The word "refund" and the phrase "money back" end up close; "refund" and "photosynthesis" end up distant. Meaning becomes distance, and distance is something a computer can measure quickly.

Nearness is scored with one of two closely related measures. Cosine similarity compares the angle between two vectors and ignores their length, so it asks purely about direction, or semantic orientation. Dot product multiplies them component by component and does factor in magnitude. When vectors are normalized to unit length, the two are equivalent, and normalized cosine similarity is the common default for text. The output is a single number, higher meaning more similar, that ranks every candidate against the query.

   semantic space (2-D sketch of a ~1024-D space)

        "money back" .   . "refund policy"
                      \  /
                       \/  <- small angle = high similarity
      query: "how do   /\
       I get a refund"/  \
                     /    \
   "reset password" .      . "photosynthesis"
        (far away = unrelated meaning)
            

Two consequences follow immediately. First, the vector is a lossy summary of the whole chunk, so an oversized or multi-topic chunk embeds as a blurred average. Second, the space is only as good as the model that built it, which is the subject of the next section.

The model sets the ceiling

Of every knob in a RAG pipeline, the embedding model has the largest effect on retrieval quality, and it acts before any downstream tuning gets a turn. Reranking, prompt engineering, and generation all operate on the candidate set that dense retrieval hands them. If the right passage never enters that candidate set because the embedding space placed it far from the query, nothing later can promote it. The model draws the ceiling; everything else works underneath it.

Four properties of a model decide whether it fits your corpus, and they trade against each other:

• Domain fit: a general-purpose embedder trained on web text may scatter specialized vocabulary (legal clauses, ICD codes, semiconductor part numbers) that a domain-tuned model keeps tightly clustered. Domain fit is usually the single biggest lever, and it is invisible until you test on your own data.
• Dimensionality: more dimensions can capture more nuance but cost more memory and compute per query and per stored vector. Some modern models support shortening the vector (Matryoshka-style) so you can trade a little accuracy for large index savings.
• Context length: the model's input limit caps how large a chunk you can embed. Feed it more and many embedders silently truncate, so text past the limit becomes invisible to search. Chunk sizing and this limit are one coupled decision.
• Multilingual reach: if your corpus or your users span languages, you need a model that maps equivalent meaning across languages into the same region, not one that clusters by language.

Public leaderboards are a starting filter, not an answer. They rank average performance across generic tasks, and your corpus is not the average. The only measurement that decides a purchase is retrieval quality on a labeled sample of your own questions and passages, which the operational section returns to. Chunk-model coupling is covered further in the chunking deep-dive.

How vector search works

Vector search is a three-step pipeline: chunk, embed, index. First the corpus is split into retrieval-sized chunks. Then each chunk is passed through the embedding model to produce its vector. Then those vectors are loaded into an index built for one job: given a query vector, return the nearest stored vectors fast. The naive way to answer that query is to compare it against every vector in the corpus (an exact brute-force scan), which is accurate but scales linearly and becomes untenable at millions of vectors and interactive latency.

Production systems therefore use approximate nearest-neighbor (ANN) indexes, which trade a small, tunable amount of recall for order-of-magnitude speedups. The two dominant families work differently. HNSW builds a layered graph of vectors and walks it greedily from coarse to fine, giving excellent recall and low latency at the cost of high memory and slower inserts. IVF partitions the space into clusters and searches only the few nearest clusters to the query, which is memory-friendly and fast to build but sensitive to how many clusters it probes. Both expose parameters that slide the recall-versus-speed trade rather than removing it.

One rule underlies the metric column: use the similarity measure the model was trained for, and normalize consistently between indexing and querying. A mismatch there silently degrades every result. Quantization (compressing vectors to fewer bits) is a further lever that shrinks the index at a small recall cost, useful once the corpus is large.

Similarity is not relevance

The most important thing to understand about embeddings is what they do not encode. They measure semantic similarity, meaning textual and topical closeness. They do not measure relevance to the user's actual intent, and they certainly do not measure factual truth. A passage can be the nearest vector to a query and be exactly the wrong answer: the same topic, the same vocabulary, the opposite fact. "The policy does cover water damage" and "the policy does not cover water damage" sit almost on top of each other in embedding space because they share nearly every word and the whole subject, yet only one is correct.

Embeddings also have a specific blind spot for exact-match and rare-term queries. Because a vector is a smoothed, distributed summary, it dilutes precisely the tokens that lexical search treats as decisive: an order number like INV-4471-B, a specific error code, a person's surname, a statute citation. Dense retrieval will happily return things that are semantically about invoices while missing the one document containing that exact identifier. This is not a tuning failure; it is inherent to how the representation works.

These two gaps, near-miss semantics and missed exact terms, are the entire reason hybrid search exists. Combining dense vectors with a lexical method such as BM25 covers the exact-match cases the vectors drop, and a reranking stage re-scores the merged candidates for genuine relevance rather than mere similarity. Vectors are a strong first-stage retriever, not a complete one.

The operational realities

The embedding space is not a configuration value you set once; it is a stored asset with a lifecycle and a cost curve. The defining operational fact is that the query vector and the stored vectors must come from the same model. Change the embedding model and every existing vector becomes incomparable to new queries, which means changing the model forces re-embedding the entire corpus. This is embedding drift, and it makes model selection a semi-permanent commitment rather than a casual experiment. A better embedder released next quarter is only an upgrade if you can afford to reprocess everything you have indexed.

Around that central constraint sit the ongoing realities of running a vector store:

1. Re-embedding cost: a model swap means running the full corpus back through inference and rebuilding the index. Budget it as a batch job whose cost scales with corpus size, and keep the raw text so it is always re-runnable.
2. Index maintenance: ANN indexes need upkeep as documents are added, updated, and deleted. HNSW graphs in particular degrade and eventually want rebuilding; stale deletes leave ghosts in results.
3. Cost at scale: memory-resident indexes for millions of high-dimensional vectors are a real infrastructure line item. Dimensionality reduction and quantization are the main levers, each trading a little recall for a lot of footprint.
4. Retrieval evaluation: you cannot manage what you do not measure. Hold a labeled set of query-to-passage pairs and trend recall@k and precision@k so every model or index change is judged on evidence, not on a demo.

The through-line is that embeddings behave like a database schema, not a hyperparameter. Treat the ingestion pipeline as versioned and re-runnable, and treat the choice of model as a decision you will live with until a measured improvement justifies the cost of reprocessing everything.

The architect view

The architect's job here is to refuse the default. The instinct to grab whatever embedding model the framework tutorial used is precisely how the retrieval ceiling gets set by accident. Because the model choice dominates quality and is expensive to reverse, it deserves a deliberate decision backed by measurement on your own corpus, made before the index is populated at scale rather than after users complain.

Three commitments separate teams whose retrieval improves over time from teams stuck relitigating the same failures:

1. Choose the embedding model on purpose. Shortlist on domain fit, context length, dimensionality, and multilingual reach, then decide with numbers, not a leaderboard rank or a blog post.
2. Evaluate retrieval on your own data. Build a golden set of real questions labeled with the passages that answer them, and make recall@k the gate every embedding or index change must pass.
3. Plan for hybrid from the start. Assume dense retrieval will need lexical search beside it and a reranker above it. Architect the merge and rerank stages in early rather than bolting them on after the near-miss and exact-match failures arrive.

Do those three things and embeddings stop being a black box and become what they should be: a measured, deliberately chosen substrate that you can reason about and improve. The next steps in the same substrate are hybrid search and reranking and, for corpora where relationships between entities carry the answer, GraphRAG. You can watch embedding geometry directly in the Embeddings Atlas in the Lab.