---
title: "How to build a RAG pipeline"
excerpt: "An end-to-end walkthrough of a production RAG system: ingestion, chunking, embeddings, vector store, retrieval, reranking, generation, and the eval loop."
date: "2026-07-09"
lastModified: "2026-07-09"
summary: "A RAG pipeline is seven stages — ingest, chunk, embed, store, retrieve, rerank, generate — wrapped in an evaluation loop. Most teams get the first five working in a weekend and spend the next month on retrieval quality and evals, which is where the system actually lives or dies."
keyTakeaways:
  - "The retriever, not the model, decides whether RAG works. Bad context in means wrong answer out, no matter how good the LLM is."
  - "Chunking and reranking are the two highest-leverage knobs. Tune them before you touch the prompt."
  - "Without an eval loop you are guessing. A small golden set turns 'feels better' into a number you can gate on."
author: "Teo Deleanu"
authorAvatar: "/team/teo.jpg"
tags: ["RAG", "AI Engineering", "Retrieval", "Production AI"]
keywords:
  - "how to build a rag pipeline"
  - "rag pipeline tutorial"
  - "retrieval augmented generation architecture"
  - "production rag system"
---

[Retrieval-augmented generation](/glossary/retrieval-augmented-generation) is the default architecture for putting an LLM to work on your own data. Instead of fine-tuning knowledge into a model, you retrieve the relevant passages at query time and hand them to the model as context. This guide walks the full pipeline end to end, in the order you should build and debug it.

## The seven stages

A RAG pipeline is seven stages: **ingest → chunk → embed → store → retrieve → rerank → generate**, wrapped in an **eval loop**. The first five get you a demo. The last two, plus the eval loop, get you something you can put in front of users.

## 1. Ingestion

Start by pulling source documents into one place and normalizing them. PDFs, HTML, Confluence pages, Slack exports — each arrives in its own format and each needs cleaning. Strip navigation chrome, boilerplate footers, and duplicate headers. Preserve structure that carries meaning: headings, lists, tables, code blocks. That structure is what makes the next stage work.

Attach metadata now, while you still have it: source URL, document title, section heading, last-modified date, access-control tags. You will use every one of these later — for filtering, for citations, and for freshness.

## 2. Chunking

[Chunking](/glossary/chunking) splits each document into passages small enough to embed and retrieve precisely. This is the single most under-appreciated stage. Split on a blind character count and you cut sentences and tables in half, producing fragments that no query matches well.

Chunk along the document's natural structure — sections and paragraphs — and add a small overlap so a fact near a boundary survives in both neighboring chunks. There's a full treatment in [how to chunk documents for RAG](/guides/how-to-chunk-documents-for-rag); for now, the rule is: the chunk is the atom of your system, and a bad atom poisons everything downstream.

## 3. Embeddings

[Embeddings](/glossary/embeddings) turn each chunk into a vector — a list of numbers that encodes meaning, so that semantically similar passages land near each other in vector space. Pick one embedding model and use it for both your chunks and your queries; mixing models breaks the geometry.

```python
from openai import OpenAI

client = OpenAI()

def embed(texts: list[str]) -> list[list[float]]:
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return [d.embedding for d in resp.data]
```

Batch your embedding calls — a few hundred chunks per request — to stay under rate limits and cut cost. Store the model name alongside the vectors so a future model swap is a clean re-index, not a silent quality regression.

## 4. Vector store

A [vector database](/glossary/vector-database) holds your embeddings and answers nearest-neighbor queries fast. For a first system, `pgvector` on the Postgres you already run is usually the right call — one fewer service to operate, and transactional consistency with your metadata for free. Reach for a dedicated store when scale or specific index features force the issue. The tradeoffs are laid out in [pgvector vs Pinecone](/compare/pgvector-vs-pinecone) and [Pinecone vs Weaviate vs Qdrant](/compare/pinecone-vs-weaviate-vs-qdrant).

Whatever you choose, store the chunk text and its metadata next to the vector so retrieval returns everything the generation stage needs in one round trip.

## 5. Retrieval

At query time, embed the user's question with the same model, then ask the store for the top-k nearest chunks. Start with `k` around 10 to 20 — wider than you think you need, because the next stage will trim it.

Pure vector search misses exact-match terms: product codes, error strings, proper nouns. **Hybrid search** — combining vector similarity with keyword (BM25) search — fixes this and is worth wiring in early. Use your metadata to filter, too: restrict to the current tenant, drop stale documents, respect access control. A retriever that returns a chunk the user isn't allowed to see is a security bug, not a relevance one.

## 6. Reranking

[Reranking](/glossary/reranking) is the highest-leverage quality lever after chunking. Your retriever optimizes for recall — casting a wide net cheaply. A cross-encoder reranker then reads each candidate chunk *together with* the query and scores true relevance, so you can feed the model the best three instead of a noisy twenty.

```python
scored = reranker.rank(query, candidates)      # cross-encoder relevance
context = [c.text for c in scored[:3]]          # keep the best few
```

This two-stage retrieve-then-rerank shape — cheap-and-wide, then expensive-and-precise — is what separates a demo from a system that answers correctly under real queries.

## 7. Generation

Now assemble the prompt: the user's question, the reranked chunks as clearly-delimited context, and an instruction to answer *only* from that context and to say "I don't know" when the answer isn't there. Grounding the model this way is the single biggest lever against [hallucination](/glossary/hallucination); there's more in [how to reduce LLM hallucinations](/guides/how-to-reduce-llm-hallucinations).

Return citations. Because each chunk carries its source metadata, you can show the user exactly which passage backed each claim — which is both a trust feature and your fastest debugging tool when an answer looks wrong.

## The eval loop

Everything above is guesswork until you measure it. Build a small **golden set** — thirty to a hundred real questions with known-good answers and the chunks that should have been retrieved. Then measure two things separately:

- **Retrieval quality**: did the right chunk make it into the context? (recall@k, hit rate)
- **Answer quality**: given good context, was the answer correct and grounded?

Separating these tells you *where* a failure lives. If retrieval missed the chunk, no prompt tweak will save you — fix chunking or reranking. If retrieval nailed it but the answer was wrong, the problem is generation. Wire this into CI so a change that quietly regresses retrieval fails the build. [How to evaluate LLMs](/guides/how-to-evaluate-llms) covers the full evaluation setup, including [LLM-as-a-judge](/glossary/llm-as-a-judge) for grading open-ended answers.

## Honest tradeoffs

RAG is not free. You are now operating an ingestion pipeline, a vector store, and an embedding model, and every one of them can drift. Re-indexing on a model change is real work. Latency adds up across retrieve, rerank, and generate — budget for it. And RAG only answers from what you retrieved; it will not reason across your whole corpus the way a human who read all of it could.

But for grounding an LLM in current, private, and citable data, nothing else touches it. Build the seven stages, wrap them in the eval loop, and tune chunking and reranking first — that's where the wins are.