Embeddings & Vector Search Fundamentals
An embedding model maps text to a dense vector, typically 384 to 3,072 dimensions, positioned so that semantically similar passages land close together under cosine similarity. At query time you embed the question, then search the corpus for nearest neighbors; at enterprise scale (millions of chunks) exact search is too slow, so vector databases use approximate nearest neighbor indexes, most commonly HNSW, a layered graph that trades a small recall loss (often tuned to 95 to 99 percent) for millisecond lookups. Model choice matters more than database choice: embeddings are not interchangeable, a corpus embedded with one model cannot be queried with another, and swapping models means re-embedding everything, so treat the embedding model as a versioned architectural commitment. Dimension count drives storage and memory cost roughly linearly; many teams over-buy dimensions when a well-chosen smaller model plus a re-ranker delivers better retrieval per dollar.
Chunking Strategies That Preserve Meaning
Chunking splits documents into retrievable units, and it quietly determines the ceiling of RAG quality: the retriever can only surface what the chunker kept intact. Fixed-size chunking (often 256 to 1,024 tokens with 10 to 20 percent overlap) is cheap and predictable but happily severs a table from its caption or a clause from its definitions. Structure-aware chunking splits on document semantics instead: headings, sections, list boundaries, table rows, and prepends the heading path to each chunk so "Termination" from an employment policy is distinguishable from "Termination" in a network runbook. Semantic chunking goes further, breaking where embedding similarity between adjacent sentences drops. The core trade-off is chunk size: small chunks improve precision and embedding sharpness but strip context; large chunks improve recall of self-contained answers but dilute the vector and burn context window. In practice, structure-aware splitting with modest overlap beats clever algorithms applied to badly parsed documents.
Hybrid Search & Re-Ranking
Hybrid search runs lexical and semantic retrieval in parallel and fuses the results, because each fails differently: BM25 nails exact identifiers, part numbers and rare jargon that embeddings smear, while dense vectors catch paraphrases BM25 cannot see. The standard fusion is reciprocal rank fusion, which merges ranked lists by rank position rather than incomparable scores, so no score normalization gymnastics are required. The fused candidates (often the top 50 to 150) then go to a cross-encoder re-ranker, a model that reads query and passage together and produces a much sharper relevance score than any bi-encoder can, at the cost of tens of milliseconds per candidate. Metadata filtering (department, date range, security label) should be applied inside the index as a pre-filter, not after retrieval, or your top-k evaporates to nothing for narrow filters. For most enterprise corpora, adding BM25 plus a re-ranker to a pure vector setup is the single highest-return retrieval upgrade available.
Grounding, Citations & Hallucination Control
Grounding is the contract that every claim in a generated answer must trace back to a retrieved chunk, and citations are how you make that contract auditable. Mechanically, the pipeline passes chunk identifiers alongside the text, instructs the model to cite them inline, and renders each citation as a link to the exact passage (with the offset highlighted) so a reviewer can verify in one click rather than trusting the summary. The harder discipline is refusal: when top retrieval scores are weak or the retrieved set does not actually contain the answer, the system should say so instead of letting the model improvise from parametric memory, which is where the most damaging hallucinations originate, fluent, plausible and unsourced. Enterprises tend to over-invest in prompt phrasing and under-invest in the retrieval threshold and the answerability check; a well-calibrated refusal is a feature that compliance teams will explicitly test for.
GraphRAG & Knowledge Graphs
GraphRAG builds a knowledge graph over the corpus, extracting entities and relations with an LLM at ingestion, then retrieves through the graph rather than by vector similarity alone. Microsoft's reference approach clusters the graph into communities (typically via Leiden), pre-summarizes each community, and answers broad questions by map-reducing over those summaries. This fixes the two question types vanilla RAG reliably fumbles: multi-hop queries ("which suppliers of our acquired subsidiaries are under sanctions?") where the answer spans documents no single chunk connects, and corpus-level questions ("what are the main risk themes across these 4,000 contracts?") where similarity search retrieves ten arbitrary fragments. The cost is real: ingestion runs one or more LLM calls per chunk, so indexing can cost orders of magnitude more than plain embedding, and extraction quality degrades on noisy source text. The architect's move is scoping: apply GraphRAG to the high-value, entity-dense slice of the corpus, not everything.
RAG vs Long Context vs Fine-Tuning
RAG, long-context stuffing and fine-tuning solve different problems, and the recurring enterprise mistake is treating them as substitutes. The decision framework runs on freshness, provenance, cost and data volume. Freshness: RAG reflects a document the moment it is re-indexed; fine-tuning bakes knowledge in at training time and goes stale immediately. Provenance: RAG can cite its sources, which matters wherever answers face audit; a fine-tuned model cannot tell you where a claim came from. Cost and volume: million-token context windows are genuinely useful for a single contract or codebase slice, but attention over long inputs degrades (the well-documented middle-of-context blind spot), and paying for hundreds of thousands of input tokens per query is ruinous at scale, even with prompt caching. Fine-tuning earns its keep for style, format and task behavior, not for facts. The pragmatic default: RAG for knowledge, long context for depth on a small working set, fine-tuning for behavior, frequently all three together.
Document Pipelines & Metadata Design
The document pipeline is the unglamorous 60 to 80 percent of a RAG program: turning PDFs, wikis, tickets and email into clean, chunked, permission-tagged text. Parsing is the first trap: enterprise PDFs mix native text with scanned pages needing OCR, and tables silently linearize into word soup unless you use layout-aware extraction that preserves cell structure (or converts tables to Markdown or HTML for the model). Every chunk needs a metadata schema designed up front: source system, document type, owner, effective date, language and, critically, the source ACL, because retrieval must honor document-level permissions at query time or the RAG system becomes a data exfiltration tool that helpfully summarizes what the user was never allowed to open. Ingestion must be incremental, driven by change detection (checksums, modified timestamps, webhooks), not periodic full re-crawls that either lag by weeks or re-embed millions of unchanged chunks at real cost.
Evaluating RAG: Faithfulness, Recall, Precision
RAG evaluation splits the system into two questions: did we retrieve the right evidence, and did the model answer only from it. Retrieval is measured with context recall (of the facts needed for a correct answer, how many appear in the retrieved chunks) and context precision (how much of what was retrieved is actually relevant, since irrelevant chunks distract the generator). Generation is measured with faithfulness or groundedness: decompose the answer into atomic claims and check each against the retrieved context, typically with an LLM judge whose own accuracy you have spot-checked against human labels. The operational backbone is a golden question set, on the order of 100 to 300 questions with expected sources and answers, drawn from real user queries and refreshed as the corpus changes; run it on every change to chunking, embeddings, prompts or models. Teams that skip this end up tuning by anecdote, and anecdote always flatters the demo.