Chunking is the least glamorous part of a retrieval-augmented generation system and the one that most often decides whether it works. Every other stage — embedding, indexing, retrieval, reranking — operates on the units you produce here. If a chunk splits a definition from its explanation, or bundles three unrelated topics into one blob, no downstream component recovers what you lost. Get this right first.
What chunking actually decides
A chunk is the atomic unit your retriever returns. That has two direct consequences. First, it sets the granularity of what the model can cite: retrieval can only ever hand back whole chunks, so if the answer to a question spans a boundary, the model sees half of it. Second, it shapes the embedding itself. An embedding is a single vector summarizing an entire chunk, so a chunk covering five topics produces a muddy average vector that matches nothing well. Tight, coherent chunks produce sharp vectors. That is the whole game.
So the goal is not "split the document into equal pieces." It is "produce self-contained units of meaning that a single query can match." The strategies below are ordered roughly by how much structure they respect.
Fixed-size chunking
The simplest approach: cut the text every N tokens (or characters), ignoring content entirely. A 512-token window with no regard for sentences or paragraphs.
It is fast, trivially parallelizable, and produces uniform chunks that play nicely with token budgets. It is also the worst at preserving meaning. Fixed-size splits cut mid-sentence and mid-word constantly, orphaning the second half of an idea. Use it only as a baseline or for genuinely unstructured text (logs, transcripts without punctuation) where nothing better applies.
Recursive / character-splitter chunking
This is the sensible default for most projects. A recursive splitter tries to break on the most meaningful separator first — paragraphs — and only falls back to finer ones (sentences, then words, then characters) when a piece is still too big. You get chunks that respect natural boundaries while staying under a size cap.
def recursive_split(text, max_len, separators=["\n\n", "\n", ". ", " "]):
if len(text) <= max_len or not separators:
return [text]
sep, rest = separators[0], separators[1:]
chunks, buf = [], ""
for part in text.split(sep):
piece = part + sep
if len(buf) + len(piece) <= max_len:
buf += piece
else:
if buf:
chunks.append(buf)
# part itself may still be too large -> recurse with finer separators
buf = piece if len(piece) <= max_len else ""
if len(piece) > max_len:
chunks.extend(recursive_split(part, max_len, rest))
if buf:
chunks.append(buf)
return chunks
Libraries like LangChain's RecursiveCharacterTextSplitter implement a hardened version of this. In practice, tune the separator list to your corpus — for prose, paragraph and sentence breaks; for code, function and class boundaries. Recursive splitting gets you 80% of the quality of fancier methods for a fraction of the effort, and it is where you should start.
Semantic chunking
Recursive splitting respects punctuation, not meaning. Two adjacent paragraphs might discuss completely different subtopics, and a size-based splitter will happily glue them together. Semantic chunking fixes this by embedding sentences (or small windows) and cutting where the topic shifts.
The mechanism: embed each sentence, walk through the document computing the similarity between consecutive sentences, and start a new chunk wherever similarity drops below a threshold — a semantic "seam." The result is chunks that track topic boundaries rather than character counts.
The tradeoff is cost and complexity. You pay for an extra embedding pass over every sentence before you even build your index, and the threshold is a knob that needs tuning per corpus. Semantic chunking helps most on dense, topically varied documents (research papers, wide-ranging reports) and helps little on already well-structured content. Reach for it when your retrieval evals show recursive splitting is merging unrelated ideas — not by default.
Structural / heading-aware chunking
Many documents already tell you where the boundaries are. Markdown has headings, HTML has sections, PDFs have a layout, code has functions. Structural chunking parses that structure and splits along it, so each chunk corresponds to a section the author already treated as a unit.
This is often the highest-leverage strategy when your source format carries structure, because it aligns chunks with human intent for free. A markdown-aware splitter keeps a heading and its body together and can carry the heading path (H1 > H2 > H3) forward as metadata, which is enormously useful for both retrieval and citation. When your corpus is docs, wikis, or knowledge bases, do structural splitting first and only apply recursive splitting to oversized sections that blow the size cap.
Chunk size and overlap tradeoffs
Two knobs dominate: size and overlap.
Size. Small chunks (say 128–256 tokens) produce precise embeddings and pinpoint retrieval, but they fragment context — the model may get the exact sentence and miss the surrounding qualification. Large chunks (800–1500 tokens) preserve context but blur the embedding and waste the model's context window on irrelevant text. There is no universal answer; it depends on how self-contained your information is. Fact-dense Q&A favors smaller chunks; narrative or reasoning-heavy content favors larger. Test a few sizes against real queries rather than guessing.
Overlap. Overlap repeats the last few sentences of one chunk at the start of the next, so an idea straddling a boundary survives in at least one chunk intact. A common starting point is 10–20% of chunk size. Overlap costs storage and duplicates content in results (which your dedup and reranking step should handle), but it is cheap insurance against boundary-severed answers. Zero overlap is a false economy for anything but the cleanest structural splits.
Attach metadata to every chunk
A chunk is not just text. Attach, at minimum: the source document ID, the heading path or section title, the position/index within the document, and any domain fields you'll filter on (product, version, date, access level). Store these alongside the vector so your vector database can do metadata-filtered retrieval — "only chunks from the 2026 handbook" — before or after the similarity search.
Metadata pays off three ways: it enables filtering that cuts the search space and removes stale or out-of-scope results; it lets you reconstruct citations ("from Handbook §3.2"); and it makes debugging tractable, because when a bad chunk gets retrieved you can trace exactly where it came from. Whether your index is Postgres-based or a dedicated service changes the filtering ergonomics — see pgvector vs Pinecone — but every serious store supports metadata, so use it.
Validate chunking with retrieval, not by eyeballing
The only honest way to judge a chunking strategy is to measure retrieval. Build a small evaluation set of real questions paired with the passages that should answer them. Then, for each candidate configuration — strategy, size, overlap — run retrieval and measure whether the right chunk shows up in the top-k results. Recall@k and hit rate are the metrics that matter here; they tell you directly whether your splits preserved the answers.
This turns chunking from taste into an experiment. Change one variable, re-run the eval, keep what wins. A 30-question eval set built in an afternoon will teach you more than any blog post's recommended defaults, including this one. If you're standing up the whole system, the broader how to build a RAG pipeline guide covers where this eval loop fits.
The practical path: start with structural or recursive splitting, add modest overlap, attach rich metadata, and let a retrieval eval tell you when — and only when — the added cost of semantic chunking earns its place.