Retrieval-augmented generation (RAG) is the default way to make an LLM answer from data it was never trained on. Instead of hoping the model memorized your docs, you retrieve the handful of passages that matter for the current question and hand them to the model as context.
Why it matters in production
RAG is the difference between a hallucination about your product and the model quoting your actual documentation. It also means you can update knowledge by re-indexing a document, not by fine-tuning the model — retrieval is almost always the right tool for facts, and training the wrong one. That decoupling is the whole point.
The common mistake
Teams treat RAG as a generation problem when it is almost always a retrieval problem. If the right passage never makes it into the context window, no prompt engineering saves you — the model can only reason over what it was handed. When a RAG system gives bad answers, look at what got retrieved before you touch the prompt. Nine times out of ten the retriever pulled the wrong chunks, or the chunking split a fact across two pieces that never travel together.
How we actually use it
We build retrieval first and generation last: get relevant embeddings into a vector database, add a reranking pass to sharpen the top results, then measure retrieval quality with evals before anyone judges the final answer. See our full walkthrough of how RAG works in production and where it hits its ceiling versus agentic RAG and big-context approaches.