---
title: "How to fine-tune an LLM"
excerpt: "A practitioner's guide to deciding when to fine-tune, using LoRA and QLoRA, preparing data, evaluating results, and serving adapters in production."
date: "2026-07-09"
lastModified: "2026-07-09"
summary: "Fine-tuning teaches a model behavior, format, and style — not facts. This guide covers the decision tree, LoRA/QLoRA, data prep, evaluation, and serving costs."
keyTakeaways:
  - "Reach for prompting first, RAG for knowledge, and fine-tuning only for behavior, format, or style you can't get any other way."
  - "LoRA and QLoRA let you adapt a large model on a single GPU by training small adapter weights instead of the whole network."
  - "A few hundred clean, well-formatted examples beat tens of thousands of noisy ones — and a held-out eval set is non-negotiable."
author: "Teo Deleanu"
authorAvatar: "/team/teo.jpg"
tags: ["Fine-tuning", "LoRA", "AI Engineering", "Training"]
keywords:
  - "how to fine-tune an llm"
  - "lora qlora fine-tuning"
  - "fine-tune vs rag"
  - "llm fine-tuning data preparation"
  - "serving lora adapters vllm"
---

Most teams that ask us to fine-tune a model don't need to. Fine-tuning is the last tool you reach for, not the first, and reaching for it early is one of the most expensive mistakes in applied AI. Start by being honest about what problem you're actually solving.

## The decision tree: prompt, then RAG, then fine-tune

Work through these in order and stop at the first one that solves your problem.

**Prompt engineering first.** Most quality problems are prompt problems. A clearer instruction, a few in-context examples, a better output schema, or a larger [context window](/glossary/context-window) fixes more than people expect. It costs nothing to try, ships in minutes, and is trivial to revert. If you haven't spent real effort here, you're not ready to fine-tune.

**RAG for knowledge.** If the model lacks *facts* — your product docs, a customer's account history, last week's changelog — the answer is [retrieval-augmented generation](/glossary/retrieval-augmented-generation), not fine-tuning. Fine-tuning is a bad way to inject knowledge: it's slow to update, you can't cite sources, and the model will confidently hallucinate around the edges of what it half-memorized. Put the knowledge in a retrieval layer and let the model read it at inference time. Our [RAG pipeline guide](/guides/how-to-build-a-rag-pipeline) covers this end to end.

**Fine-tune for behavior.** This is the one thing the other two can't do well. [Fine-tuning](/glossary/fine-tuning) teaches a model *how to behave* — a consistent output format, a house tone of voice, a domain-specific reasoning pattern, adherence to a rigid schema, or a skill that's awkward to specify in a prompt. Say it out loud: **fine-tuning teaches behavior, not facts.** If your failing test cases are about *what* the model knows, fine-tuning won't help. If they're about *how* it responds, it might.

A useful tell: if you can fix a bad output by pasting three good examples into the prompt, you have a prompting or RAG problem. If you'd need to paste fifty examples every single call to get consistent behavior, that's a fine-tuning candidate — you're paying to bake those examples into the weights.

## Full fine-tuning vs LoRA vs QLoRA

Once you've decided to fine-tune, the next choice is *how much of the model to touch*.

**Full fine-tuning** updates every weight in the network. It's the most powerful option and almost never the right one for a small team. You need enough GPU memory to hold the model, its gradients, and optimizer states — several times the model's inference footprint — and you produce a full-size checkpoint per experiment. For a 7B+ model this gets expensive fast.

**LoRA (Low-Rank Adaptation)** freezes the original weights and trains small "adapter" matrices injected alongside them. The insight is that the *update* to a weight matrix during fine-tuning tends to be low-rank, so you can approximate it with two thin matrices whose product has the same shape. You train maybe a fraction of a percent of the parameters, the base model stays frozen, and your output is a tiny adapter file instead of a multi-gigabyte checkpoint. This is the default choice for most production work.

**QLoRA** goes further: it quantizes the frozen base model to 4-bit and trains LoRA adapters on top of it. That collapses the memory needed to *hold* the base model, so you can fine-tune models that otherwise wouldn't fit — often on a single consumer or mid-tier GPU. You pay a little in training speed and a little in precision, but for most tasks the quality difference is negligible. If you're memory-constrained, start here.

A typical LoRA config with Hugging Face PEFT:

```python
from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                # rank of the adapter matrices
    lora_alpha=32,       # scaling; a common rule is alpha = 2 * r
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()  # sanity-check: should be a small %
```

`r` (the rank) is your main knob. Higher rank means more capacity to learn but more parameters and more risk of overfitting on a small dataset. Rank 8–32 covers most cases; start low and only raise it if the model is clearly underfitting.

## Data preparation: quality over quantity

This is where projects are won or lost. The model learns the distribution you show it, including your mistakes. A few hundred hand-checked examples routinely beat tens of thousands of scraped, inconsistent ones.

Format your data as instruction/response pairs — usually JSONL, one example per line:

```jsonl
{"messages": [{"role": "system", "content": "You are a support agent. Reply in JSON with keys `intent` and `reply`."}, {"role": "user", "content": "My invoice is wrong, I was charged twice."}, {"role": "assistant", "content": "{\"intent\": \"billing_dispute\", \"reply\": \"I'm sorry about the double charge. I've flagged it for a refund, which lands in 3-5 business days.\"}"}]}
{"messages": [{"role": "system", "content": "You are a support agent. Reply in JSON with keys `intent` and `reply`."}, {"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "{\"intent\": \"account_help\", \"reply\": \"Head to Settings > Security and click Reset Password. You'll get an email link within a minute.\"}"}]}
```

Rules that matter more than volume:

- **Match production exactly.** The prompt format, system message, and output schema in your training data must be identical to what you'll send at inference. A mismatch quietly destroys quality.
- **Diversify the inputs, standardize the outputs.** Cover the real spread of user phrasing, but keep the *shape* of good answers consistent — that consistency is the behavior you're teaching.
- **Curate, don't just collect.** Every bad example is a lesson in bad behavior. Read your data. Fix or drop the wrong ones.
- **Hold out a test set before you touch training.** Split off 10–20% of examples the model never sees. Without this you cannot know if you improved anything.

Note that these are text pairs, not [embeddings](/glossary/embeddings) — fine-tuning changes the model's weights, whereas embeddings power the retrieval half of a RAG system.

## Running the tune and evaluating it

The training loop itself is the easy part — frameworks like `trl`'s `SFTTrainer`, Axolotl, or Unsloth wrap it in a config file. Train for a small number of epochs (often 1–3); watch validation loss and stop when it stops falling, because LoRA on a small dataset overfits quickly. Keep the learning rate modest.

The hard part is knowing whether it worked. Training loss going down is *not* proof of anything useful. Evaluate the fine-tuned model against your held-out set and against the un-tuned base model on the same set, using metrics that reflect the actual task: exact-match or schema-validity for structured output, an LLM-as-judge rubric for tone and quality, task-specific accuracy for classification. Build this into a repeatable harness — our [LLM evaluation guide](/guides/how-to-evaluate-llms) and the [evals glossary entry](/glossary/llm-evals) go deep on this. If the tuned model doesn't clearly beat the base model on your own eval, you don't have a result yet.

## Cost and serving

Training cost is usually the smaller number. LoRA and especially QLoRA keep it low — often a few GPU-hours for a small dataset. The ongoing cost is *serving*.

The nice property of LoRA is that adapters are tiny and hot-swappable. A serving stack like vLLM can hold one base model in GPU memory and serve *many* LoRA adapters on top of it, routing each request to the right one. That means you can ship per-customer or per-task fine-tunes without paying for a separate GPU per model — a huge economic advantage over full fine-tuning, where each variant is a full-weight model you must host separately. If you're heading to production, see our guide on [deploying vLLM in production](/guides/how-to-deploy-vllm-in-production) for the serving details, and [vLLM vs TGI](/compare/vllm-vs-tgi) for picking the inference engine.

The honest bottom line: fine-tune when you've proven that prompting and RAG can't produce the *behavior* you need, use LoRA or QLoRA to keep it cheap and swappable, invest most of your effort in clean data and a real eval set, and measure against the base model before you call it a win.