Chunking: the Most Underrated Decision in Your RAG Pipeline

Why chunking decides retrieval quality

Every RAG pipeline contains one decision made once, early, usually by whoever wrote the ingestion script, that silently caps everything built on top of it: how documents get cut into chunks. The retriever does not search your documents. It searches your chunks. If the answer to a user's question was split across two chunks, or buried in a chunk that is mostly about something else, the best embedding model and the best re-ranker in the world are competing to return the wrong text fragment slightly faster.

The failure is mechanical and worth stating precisely. An embedding vector is a fixed-size summary of everything in the chunk. Oversize the chunk and the vector becomes an average of several topics; a pointed query about one clause in a 3,000-token chunk matches weakly because the clause is diluted by everything around it. Undersize the chunk and you orphan context: a fragment reading "the penalty in this case is doubled" embeds fine, retrieves fine, and is useless, because "this case" was defined two paragraphs earlier in a chunk that did not come along. Both failures happen before the language model ever runs.

This is also why chunking problems masquerade as model problems. The symptom reaches you as a hallucinated answer or an unhelpful "I could not find that", and the instinctive fixes (a bigger model, a longer prompt, a re-ranking stage) all operate downstream of the damage. Diagnosing where RAG failures actually live is covered in Evaluating RAG; the short version is that a large share of them trace back to retrieval, and retrieval traces back to what the splitter did to the corpus.

Fixed, recursive, semantic: the strategy ladder

Chunking strategies form a ladder from trivial to expensive, and the right rung depends on the corpus, not on ambition. The baseline is fixed-size splitting with overlap: cut every N tokens, slide the window so consecutive chunks share an overlap region. It is fast, deterministic, and blind. It will happily cut a sentence in half, separate a table header from its rows, and split a contract clause across a chunk boundary. It remains the right choice for exactly one situation: an unstructured, homogeneous corpus you need indexed today, with a plan to revisit.

One rung up, recursive (separator-based) splitting tries to break on natural boundaries in priority order: section breaks first, then paragraphs, then sentences, then words, only descending when a piece is still too large. For prose, this fixes the ugliest fixed-size failures at essentially zero extra cost, which is why it is the default in most frameworks and a sensible default in practice. What it still lacks is any notion of document semantics: it sees separator characters, not headings, lists or tables.

Semantic chunking goes further: embed each sentence, walk the document, and place a boundary where similarity between adjacent sentences drops, so each chunk covers one coherent topic. It costs an embedding pass over the entire corpus at ingestion, its boundaries shift when you change embedding models, and in our experience it earns its cost mainly on long, meandering prose without reliable formatting, think transcripts and email threads, rather than well-structured documents.

Structure-aware chunking

The highest-return rung for most enterprise corpora is the one that respects how the document was written. Authors already segmented their documents: headings, numbered clauses, list items, table boundaries. A structure-aware splitter parses that markup (Markdown and HTML-aware splitters are standard; PDF requires a layout-extraction step first, covered in Document Pipelines) and refuses to place a boundary inside an element. The rules that matter most: never separate a heading from its first paragraph, never split a list mid-item, and never, under any budget pressure, bisect a table. A table sliced horizontally loses its header row; the orphaned half is rows of numbers with no column names, which is worse than not indexing it at all. Keep tables whole, keep their captions attached, and if a table exceeds the chunk budget, split by row groups and repeat the header row in every piece.

Structure awareness also unlocks the most useful retrieval pattern in production RAG: parent-child chunking. The tension it resolves is real: small chunks embed precisely and match queries well, but arrive at the generator stripped of surrounding context. So index small and deliver large. Embed fine-grained child chunks for matching; when a child matches, return its parent (the enclosing section) to the model.

 DOCUMENT
    |
    +-- parent: full section (~1,500 tokens, stored)
           |
           +-- child 1 (~200 tokens) --> embedded + indexed
           +-- child 2 (~200 tokens) --> embedded + indexed
           +-- child 3 (~200 tokens) --> embedded + indexed
           +-- ...      (children tile the full section)

 query matches child 2 --> parent section goes to the LLM
            

The cost is a slightly more involved index (children carry a pointer to their parent) and deduplication when several children of the same parent match. Both are solved problems in mainstream retrieval stacks, and the quality gain on heading-structured corpora is typically the largest single win available.

Sizes, overlap and the numbers that matter

The numbers are less mysterious than the volume of debate suggests. Typical production chunk sizes run from roughly 200 to 800 tokens, and the spread is driven by content type. Dense, self-contained material (FAQ entries, API references, policy clauses) works best small, on the order of 200 to 400 tokens, because each unit already stands alone. Narrative prose (reports, wikis, long-form documentation) tolerates and often needs 400 to 800 tokens so that an argument is not amputated mid-development. Below roughly 100 tokens, chunks stop carrying enough signal to embed distinctively; beyond roughly 1,000, dilution sets in and precision decays.

Overlap between consecutive chunks typically lands at 10 to 20 percent of chunk size. Its job is to ensure a sentence near a boundary appears whole in at least one chunk; it is insurance against boundary placement, not a substitute for good boundaries. Push overlap much past 25 percent and you are mostly paying to embed and store the same text twice while near-duplicate chunks crowd each other out of the top-k. One hard constraint sits underneath all of this: the embedding model's input limit. Many widely deployed embedders truncate silently past their limit (historically often around 512 tokens, with long-context embedders now common); a chunk that exceeds it gets embedded as its own opening paragraph, and everything after the cutoff becomes invisible to search. Check the limit, then budget chunks below it. See Embeddings & Vector Search for model selection.

Finally, treat metadata as a first-class output of chunking, not an afterthought. Every chunk should leave the splitter carrying at least:

• Source identity: document URI, title, and version or last-modified date, so answers can cite and staleness can be filtered.
• Structural position: the heading path ("Chapter 4 > Termination > 4.2"), page or section number, and parent-chunk pointer.
• Access control: the source system's ACL or entitlement tag, enforced as a filter at query time, never left to the prompt.
• Content signals: language, content type (prose, table, code), and anything your filters will need later.

Contextual enrichment and late chunking

Even a perfectly placed boundary loses one thing: the chunk no longer knows what document it came from. A chunk reading "revenue grew 3 percent over the prior quarter" is stranded; which company, which year, which segment all lived elsewhere in the document. Contextual retrieval, described by Anthropic in September 2024, attacks this directly: at ingestion, an LLM reads the full document and writes a short, chunk-specific context blurb (which company, which section, what period), which is prepended to the chunk before embedding and BM25 indexing. The reported effect was a substantial reduction in retrieval failure rates, on the order of tens of percent relative, larger still when combined with hybrid search and re-ranking. The cost is a one-time generation pass per chunk at ingestion, made materially cheaper by prompt caching since the full document is reused across all of its chunks. For corpora where retrieval quality carries business risk, it is one of the best-evidenced upgrades available.

Late chunking (introduced by Jina AI in 2024) reaches a similar goal by reordering operations. Instead of splitting first and embedding fragments in isolation, a long-context embedder processes a large span of the document in one pass, so every token's representation is conditioned on the surrounding text; the pipeline then pools token vectors per chunk afterward. Each chunk vector carries document-level context without any generation step. It requires an embedder that supports the pattern and long inputs, and it enriches only the dense vectors, not the chunk text a BM25 index or the generator sees.

A third pattern, summaries-as-chunks, helps with questions no passage answers alone ("what is this contract's overall risk posture"): index an LLM-written summary of each document or major section as an additional retrievable chunk, pointing back to the full text.

Evaluating a chunking change

Most teams change chunking the way they would never change a model: edit the splitter config, re-index, ask three questions in a demo UI, and call it improved. The alternative costs about a week once and pays for itself on every subsequent change. Build a golden set of question-to-passage pairs: 100 to 300 real user questions (pulled from logs where possible), each labeled with the document passages that answer it. Labels point at source passages, not at chunks, precisely so the set survives re-chunking; a chunk counts as relevant if it overlaps a labeled passage.

With that in hand, retrieval quality becomes two numbers you can trend: recall@k (of the labeled passages, how many surfaced in the top k) and precision@k (of the k returned chunks, how many were relevant). Run them before and after any chunking change, and sweep chunk size like the hyperparameter it is: index the corpus at, say, 200, 400 and 800 tokens with 15 percent overlap and let the golden set pick the winner per corpus. The sweep's marginal cost is re-embedding, which for most corpora is a batch job priced in single-digit dollars per gigabyte of text, illustratively, not a research program. The mechanics slot into the same CI gating discipline as any eval; see Evaluating RAG, and the RAG Explorer in the Lab lets you watch boundaries move as parameters change.

When a query fails, classify it. A small failure taxonomy turns anecdotes into a fixable backlog:

• Split answers: the answer spans a boundary and no single chunk contains it. Fix: larger chunks, more overlap, or parent-child.
• Orphan pronouns: the matched chunk says "it" or "this agreement" with the antecedent elsewhere. Fix: contextual enrichment.
• Lost table headers: numbers retrieved without column names. Fix: table isolation with header repetition.
• Diluted chunks: the right passage sits inside an oversized, multi-topic chunk that never ranks. Fix: smaller children, structure-aware boundaries.

A decision guide by corpus type

Chunking strategy should be chosen per corpus, not per platform. A pipeline that runs one splitter configuration across contracts, wiki pages and call transcripts is optimizing for none of them, and the ingestion layer should treat the splitter as a per-source policy, routed by content type. The recommendations below reflect what has repeatedly worked in enterprise settings; treat them as defensible starting points to sweep from, not endpoints.

Two closing observations for the architect who owns this decision. First, chunking is cheap to change and expensive to ignore: re-chunking and re-embedding a corpus is a batch job, while a quarter of degraded answer quality is a trust problem with users that no batch job repairs. Keep the raw documents and run ingestion as a versioned, re-runnable pipeline so the splitter is never a one-way door. Second, the discipline compounds. A team that holds a golden set, sweeps sizes per corpus, and classifies retrieval failures will make every subsequent decision (embedder swaps, hybrid search and re-ranking, enrichment) with evidence instead of folklore. The splitter is fifty lines of configuration. What it deserves is the same respect you give the model.