Reranking is a second opinion on your search results. The first stage — embedding similarity search over a vector database — is fast but coarse; it casts a wide net. A reranker takes that net, looks at each candidate against the query more carefully, and reorders so the genuinely-relevant passages land at the top.
Why it matters in production
Vector search optimizes for speed at scale, which means it approximates. It reliably gets the right passage into the top 50 but not always into the top 5 — and the top 5 is what fits the context window. A reranker closes that gap. For many mediocre RAG systems, adding a rerank pass is the single cheapest accuracy win available, far cheaper than swapping databases or fine-tuning.
The common mistake
Skipping reranking and blaming the answer. When a RAG system retrieves the right document but ranks it eighth, and you only pass the top three to the model, the model never sees it — and the failure looks like a generation problem when it is a ranking one. Retrieve broad, rank sharp: the two stages do different jobs.
How we actually use it
We retrieve a generous candidate set with fast vector search, then rerank down to the few passages that actually make it into context. That two-stage shape — broad recall, precise ranking — is the backbone of how RAG works in production, and we measure the top-k retrieval quality with evals so we know the rerank is earning its latency.