---
title: 'RAG vs Fine-Tuning: Which One Your Problem Actually Needs'
excerpt: 'The two get pitched as rivals. They solve different problems — knowledge freshness vs behavior shaping. How to pick, from systems we run.'
date: '2026-07-14'
lastModified: '2026-07-14'
author: 'Teo Deleanu'
authorAvatar: '/team/teo.jpg'
tags: ['Production AI', 'AI Engineering', 'RAG', 'Fine-Tuning']
keywords: ['rag vs fine-tuning', 'rag or fine tuning', 'when to fine tune llm']
options: ['RAG', 'Fine-tuning']
verdict: 'Default to RAG for knowledge problems and prompt engineering for behavior; fine-tune only when you have evals proving the cheaper options plateaued.'
featured: true
tldr: 'RAG fixes what the model knows; fine-tuning fixes how the model behaves. Most teams asking this question have a knowledge problem.'
keyTakeaways:
  - 'RAG: fresh/private knowledge, per-query cost, no training pipeline'
  - 'Fine-tuning: consistent style/format/behavior, needs data + eval discipline'
  - 'They compose — the question is sequencing, not either/or'
faqs:
  - q: 'Is fine-tuning cheaper than RAG at scale?'
    a: 'They spend money in different places: fine-tuning front-loads cost into training and data curation, RAG pays per query in retrieval and context tokens. Which wins depends on query volume and how often your knowledge changes — frequently-changing knowledge makes fine-tuning a treadmill.'
  - q: 'Can I use both together?'
    a: 'Yes, and mature systems often do: fine-tune (or prompt-engineer) for output behavior and format, retrieve for facts. Do them in that order of proof — behavior via prompts first, retrieval second, training last.'
  - q: 'When is fine-tuning clearly the right call?'
    a: 'When you need consistent behavior the base model will not hold via prompting — strict output formats, a specific voice, domain-specific reasoning patterns — and you have enough quality examples plus evals to measure the gain.'
---

"Should we fine-tune or do RAG?" is the wrong question, in the same way "should we buy a database or write documentation?" is the wrong question. The two techniques change different parts of the system. RAG changes what the model sees; fine-tuning changes what the model is. Once you frame it that way, most of the decision makes itself — and most teams asking it discover they have a knowledge problem, which is RAG territory.

## What each one actually changes

A language model at inference time is a fixed function of two inputs: its weights and its context window. Everything you can do to change its output goes through one of those two doors.

**RAG goes through the context door.** At query time, a retrieval pipeline finds the documents relevant to the question and places them in the prompt, and the model answers from what it was shown. The weights never change. The knowledge lives outside the model — in a vector index, a keyword index, usually both — which means it can be updated the moment a document changes, inspected when an answer looks wrong, and deleted when a customer invokes their data rights. The model is the last stage of the pipeline, and usually the least broken one.

**Fine-tuning goes through the weights door.** You continue training the model on your own examples until its default behavior shifts: the tone it adopts, the format it emits, the reasoning patterns it reaches for. The change is baked in — every response carries it, with no per-query token cost for instructions the model no longer needs to be told. But the knowledge cutoff problem travels with the weights: what the model learned in training is frozen at training time, and updating it means training again.

That asymmetry is the whole comparison. Knowledge changes constantly; behavior changes rarely. Putting fast-changing things behind the slow door is how teams end up on a retraining treadmill.

## The decision table

| Dimension | RAG | Fine-tuning |
| --- | --- | --- |
| Knowledge freshness | Update the index, effective immediately | Frozen at training time; retrain to update |
| Data you need | Documents you already have | Curated example pairs, enough of them, quality-controlled |
| Cost profile | Per query: retrieval + context tokens | Front-loaded: data curation + training, plus hosting a custom model |
| Latency | Retrieval step adds latency; big contexts slow generation | No retrieval hop; shorter prompts can be faster |
| Attribution | Answers cite retrieved chunks; you can audit why | Behavior is diffuse in the weights; no per-answer provenance |
| Failure mode | Retrieval misses → model summarizes the wrong chunks | Regressions and drift you can only catch with evals |
| Team maturity needed | An ingestion pipeline and retrieval metrics | All of that, plus training infrastructure and eval discipline |

The rows compound. If your knowledge changes weekly, the freshness row alone decides it. If you need to show a customer *why* the system answered what it did, the attribution row decides it. Fine-tuning wins the rows about behavior: when the requirement is "always answer in this exact schema" or "hold this voice across ten thousand tickets," no amount of retrieved context makes that guarantee, and stuffing instructions into every prompt is a per-query tax that training pays once.

## What we run, and why

Every knowledge-shaped problem in the systems we operate is solved with retrieval, not training. The [live demo on this site](/demo) runs the full pipeline — parse, chunk, embed, index into pgvector, hybrid retrieval fused with RRF, grounded generation with citations — and holds retrieval recall@k at 0.93, a number that is re-scored on every pull request. The stage-by-stage walkthrough is in [how RAG works in production](/blog/how-rag-works-in-production). A due-diligence platform we built does the same over PE/VC questionnaire corpora; a recruiting platform runs hybrid retrieval on OpenSearch. In each case the deciding factor was the table above: the corpora change constantly, the answers need citations, and the model's job is to summarize what retrieval hands it.

The other reason is operational, and it took a provider to teach it to us: hosted models retire on someone else's schedule. Google retired a model snapshot three days after one of our deploys, and the fix was a one-line config change *because* the knowledge lived in the retrieval layer and not in the weights — the story is in [surviving a model migration](/blog/surviving-a-model-migration). A fine-tuned model is welded to its base model. When that base model deprecates, your training investment deprecates with it, and the migration is a retraining project instead of a config change.

None of the systems above fine-tune, and that is itself the receipt: with a solid retrieval layer, careful prompting, and per-task model routing, we have not yet hit a behavior requirement that prompting could not hold. That is what "fine-tune when the cheaper options plateau" looks like in practice — we have not hit the plateau.

<!-- CTA -->

## The failure modes, honestly

**RAG fails at retrieval, quietly.** The model faithfully summarizes whatever chunks it was handed; if retrieval fetched the wrong ones, the answer is confidently wrong with plausible-looking citations. That is why the number to obsess over is recall, and why it needs a CI gate rather than a launch-week screenshot — the machinery for that is in [evals as CI](/blog/evals-as-ci). RAG also carries a real infrastructure bill: an ingestion pipeline, an index to operate (the [pgvector vs Pinecone comparison](/compare/pgvector-vs-pinecone) covers that choice), and per-query context tokens forever.

**Fine-tuning fails at measurement.** Training on your examples shifts everything, not just the behavior you wanted; capabilities you never tested can regress silently. Without a golden dataset and an eval harness, you cannot tell whether the fine-tune helped, hurt, or did nothing — you have a vibe and a training bill. It also fails at logistics: the technique needs enough high-quality examples to matter, and curating them is usually more work than teams budget. And it fails at freshness, structurally: every knowledge update is a training run.

Neither list is a disqualification. Both are bills, and the question is which bill matches your problem.

## Sequencing: the order of proof

The practical answer is not either/or — mature systems compose them: retrieval for facts, tuned or prompted behavior for form. The question is what order you spend money in, and the order that wastes the least is cheapest-first:

1. **Prompting first.** Most behavior requirements — format, tone, structure — yield to a well-engineered prompt and a capable base model. Free to iterate, instant to deploy, trivially reversible.
2. **RAG second**, the moment the problem involves knowledge the model was not trained on: your documents, your data, anything that changes. No prompt fixes a knowledge gap.
3. **Fine-tuning last**, when evals prove the first two plateaued: the format still breaks at some measured rate, the voice still drifts, the per-query instruction overhead is a bill worth eliminating. At that point you have the eval harness fine-tuning requires anyway — because you used it to prove the plateau.

Teams that run this sequence usually stop at step two. The ones that reach step three arrive with the data and the evals to do it properly. The ones that skip to step three arrive with neither, which is how "we fine-tuned and it got worse, we think" postmortems get written.