Chunking is how you decide the unit of retrieval. Before you embed a document, you split it into passages — chunks — and each chunk becomes one searchable vector. Get the split right and retrieval returns a coherent, self-contained fact. Get it wrong and it returns half a sentence or an unfocused blur.
Why it matters in production
Chunk size is a tradeoff you cannot avoid. Chunks too large produce vague embeddings that match many queries weakly and match none precisely. Chunks too small sever the context a fact needs — the answer is in chunk 4 but the qualifying condition was in chunk 3, and they never travel together. The chunk is the atom your whole RAG system is built from, and a bad atom poisons everything downstream.
The common mistake
Splitting on a fixed character count with no regard for structure. Cutting mid-sentence or mid-table destroys meaning, and a fact split across a naive boundary becomes unretrievable — no query matches either fragment well. The failure shows up as "the model doesn't know something that is clearly in the docs," and the cause is a boundary, not the model.
How we actually use it
We chunk along the document's natural structure — sections, headings, logical units — rather than blind character counts, and we add overlap so a fact near a boundary survives in both neighbors. Then we validate it the only way that counts: does a question we expect to answer actually retrieve the chunk that answers it? That check, part of how RAG works in production, turns chunking from guesswork into something measurable.