Fine-Tuning vs RAG: A Decision Procedure for 2026

Somebody on your team has a prompt that works about seventy per cent of the time, and somebody above them has asked: should we fine-tune? The short version of fine-tuning vs RAG is that most teams fine-tune when they should retrieve, and a quieter group retrieves when they should distil.

Both mistakes share a cause: nobody wrote down what was failing. If the model gets facts wrong, gradient descent will not reliably fix it. If it is accurate but expensive at volume, retrieval will not help. The failure mode picks the tool.

What follows is the mechanism, evidence from the primary papers, September 2026 prices, three code samples and a one-week protocol. One thing pins this to the year: OpenAI is winding its fine-tuning API down, and almost nobody writing about fine tuning vs RAG has noticed.

Fine-tuning vs RAG: what each mechanism actually changes

Fine-tuning changes the model's weights, so it reliably changes behaviour — output shape, tone, tool-call format, verbosity — and only weakly changes what the model knows. RAG changes what is in the context window at inference time, so it owns facts: freshness, provenance, deletion and per-user access control. Different mechanisms, different failure modes.

Fine-tuning continues training on your examples. Supervised fine-tuning (SFT) adjusts every weight; LoRA, or low-rank adaptation, freezes them and trains a small pair of matrices alongside. Either way the signal is next-token loss, which fits functions superbly and writes facts poorly: a fact must be seen many times, in many phrasings, before it durably moves the weights.

Retrieval-augmented generation (RAG) never touches weights. You embed documents as vectors and at query time fetch the closest chunks into the prompt, so what the model knows about your domain arrives as text it reads fresh each request. That is why RAG owns what weights cannot express: freshness, provenance, deletion and per-user access control.

A weight cannot carry a timestamp, a source URL or an access rule. A retrieved chunk cannot stop the model emitting prose when you asked for JSON. Arguing about which is better in general means you have not said what is broken in particular.

Why fine-tuning is bad at facts: the measured case

The cleanest experiment is Ovadia and colleagues at Microsoft, published at EMNLP 2024 as “Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs”. They built a 910-question task about facts postdating the models' training cutoffs, then compared base models, models fine-tuned on those documents for five epochs, and the same models given the documents through retrieval.

Read the Llama2-7B row twice: training on the documents containing the answers made the model worse at answering questions about them. Paraphrase augmentation, the standard fix when SFT fails to stick, lifted Mistral only to 0.588. On MMLU anatomy zero-shot: 0.556 base, 0.570 fine-tuned, 0.681 with retrieval.

Gekhman and colleagues at Google (EMNLP 2024) supply the mechanism. Examples containing genuinely new knowledge are learned significantly more slowly, and as the model finally fits them, its hallucination rate on facts it previously had right rises linearly. You trade a known gap for a confident, unsourced error elsewhere.

The counter-evidence is instructive. EntiGraph (Yang, Hashimoto and colleagues, ICLR 2025) turned 1.3 million real tokens into 455 million synthetic ones and continued pretraining Llama-3-8B, lifting closed-book accuracy from 39.49% to 56.42% — roughly 80% of what the documents give you at inference time.

What fine-tuning is good at: format, behaviour and cost compression

The same mechanism that makes fine-tuning bad at facts makes it excellent elsewhere. Schema adherence, house tone, refusal policy, the shape of a tool call, domain shorthand, reasoning length: all are functions of the input, and functions are what gradient descent fits well. Right content in the wrong form, and a few hundred examples beat a longer prompt.

The higher-value version is compression. Take a frontier model that already passes your eval set, generate its outputs on real traffic, and train a small open model to imitate them on that one task. LangChain's October 2023 recipe remains the cleanest public one: a few hundred GPT-4-generated summaries tuned into gpt-3.5 won 96% of pairwise comparisons against GPT-4 zero-shot, 63% cheaper and 11x faster.

Predibase's LoRA Land (February 2024, their own benchmark) trained 25 task-specific Mistral-7B adapters for under $8 each and served all of them from one A100, so a fine-tune need not own a GPU alone. Bridgewater's AIA Labs reported in July 2026 that a tuned Qwen3-235B averaged 84.7% across six financial-judgment tasks against 78.2% for the best frontier model, at 13.8x lower cost — vendor-internal and unreproduced.

The only fine-tune that reliably pays for itself in 2026 is distillation for cost and latency, not knowledge. If the pitch is “so the model knows our products”, stop. If it is “so we serve this task on an 8B model at a twentieth of the price”, keep going: that is the road to a model built around your own workflow.

When to fine-tune an LLM: five conditions

All five should be true before you spend a sprint. If one is false, the fine-tune either fails or succeeds invisibly, which is worse.

If you cannot name your held-out eval set, you are not allowed to fine-tune yet.

• A written eval set exists: at least 100 held-out items from real production traffic, graded, with a pass bar fixed beforehand.
• The failure is shape or behaviour: format, tone, tool call, verbosity, refusal policy, not missing information.
• Prompting plus caching already failed. Better prompt, five few-shot examples, cached prefix, measured again. Fine-tuning comes after that, never instead.
• You have 500 to 1,000 clean examples, or a frontier model that can generate them.
• Volume justifies the serving cost, and a regression suite exists. A dedicated endpoint bills whether or not anyone calls it.

The 2026 tooling reality: OpenAI is winding down fine-tuning

On 7 May 2026 OpenAI put fine-tuning on its deprecations page. From that date, organisations that had never run fine-tuning cannot create jobs. From 2 July 2026, organisations without inference on a tuned model in the past 60 days lost access. On 6 January 2027, existing customers can no longer create new jobs. Inference on models you already tuned stops only when the base model is deprecated.

That leaves a smaller, better-understood set, as of September 2026.

Two details matter. Anthropic offers no fine-tuning through its own API; the only Claude model tunable anywhere is Claude 3 Haiku on Amazon Bedrock in us-west-2, so check the Bedrock custom-models documentation before planning around it. And training prices are a rounding error: $0.50 per million tokens at Fireworks, from $0.34 at Together, roughly double for full fine-tuning or DPO. The serving GPU is what costs money.

Try prompt caching before you fine-tune

A large share of fine-tuning projects exist only to shorten a prompt. If yours is thirty thousand tokens of policy and examples sent on every request, the cost and latency are real. Training that into weights is one fix; prompt caching is the other, and it takes an afternoon. Anthropic's published rates: the five-minute cache write costs 1.25x base input, the one-hour write 2x, every cache read 0.1x. The minimum cacheable prefix runs 512 to 4,096 tokens by model, 1,024 for Claude Sonnet 5.

01import anthropic
02
03client = anthropic.Anthropic()
04MODEL = "claude-sonnet-5"
05IN_PER_MTOK, OUT_PER_MTOK = 3.00, 15.00   # placeholders: use your own rates
06
07SYSTEM_PROMPT = open("policy_and_examples.md").read()   # ~30k tokens, static
08
09resp = client.messages.create(
10    model=MODEL,
11    max_tokens=512,
12    system=[{
13        "type": "text",
14        "text": SYSTEM_PROMPT,
15        "cache_control": {"type": "ephemeral"},   # 5-minute window
16    }],
17    messages=[{"role": "user", "content": "Classify support ticket 8812."}],
18)
19
20u = resp.usage
21cost = (
22    u.input_tokens * IN_PER_MTOK
23    + u.cache_creation_input_tokens * IN_PER_MTOK * 1.25   # 1.25x on write
24    + u.cache_read_input_tokens * IN_PER_MTOK * 0.10       # 0.1x on read
25    + u.output_tokens * OUT_PER_MTOK
26) / 1_000_000
27
28print(
29    f"written={u.cache_creation_input_tokens} "
30    f"read={u.cache_read_input_tokens} "
31    f"cost_per_request=${cost:.5f}"
32)

The rates in that sample are placeholders. What matters is the ratio and the hit rate: if cache_read_input_tokens is large and cache_creation_input_tokens small across a day of production traffic, your long prompt is already close to free.

What RAG costs vs what a fine-tune costs

Here are the honest numbers, approximate, September 2026. Training ten thousand examples on an 8B model with QLoRA on one rented H100 takes eight to twelve hours at roughly $2 to $7 an hour, so $10 to $16 of GPU time, or a few dollars to $15 of managed training tokens. Indexing ten thousand documents costs a couple of dollars of embeddings plus $45 to $70 a month of vector storage. A dedicated serving GPU runs roughly $4,000 to $6,000 a month.

Training is not the expensive part. The expensive parts are the eval set somebody builds by hand, the endpoint that bills whether traffic arrives or not, and the retrain treadmill: every domain shift puts you back in the pipeline while the RAG team did an upsert. Our breakdown of what an AI agent project actually costs has the wider budget picture.

The strongest argument against RAG is worth knowing. The “Token Tax” paper (Hamilton and colleagues, June 2026) compared retrieval against putting whole document sets into a long context window, across 972 answers and two small models. Long context won on accuracy, 73.1% against 65.4% for semantic RAG, at roughly 26 times the per-query token cost. Retrieve to narrow, then reason long, and watch what long contexts do to accuracy, covered in our piece on context rot.

RAFT: when fine-tuning and RAG together wins

One fine-tuning recipe unambiguously beats both alternatives, and teams skip it. RAFT, from Zhang, Patil and colleagues in Berkeley's Gorilla group in 2024, tunes the model to read retrieved chunks including deliberately wrong ones, and to quote the span it used before answering. On a LLaMA2-7B base their PubMed numbers run 56.5, then 58.8 with RAG, 59.7 with domain fine-tuning, and 73.3 with RAFT. On the HuggingFace API task: 0.22 to 74.0.

Use it when retrieval already works, recall@10 above roughly 85%, but the model mishandles what it gets: ignores the right chunk, falls for a near miss, or answers from memory. The format is the method: one oracle chunk, k distractors, a deliberate fraction with no oracle, and an answer that quotes verbatim before concluding.

01import json
02import random
03
04
05def raft_example(question, oracle, distractors, final, quote, k=4, p_oracle=0.8):
06    """One RAFT-style training row: oracle sometimes absent, answer quotes first."""
07    assert quote in oracle, "the cited span must appear verbatim in the oracle chunk"
08    keep_oracle = random.random() < p_oracle
09
10    chunks = random.sample(distractors, k if keep_oracle else k + 1)
11    if keep_oracle:
12        chunks.append(oracle)
13    random.shuffle(chunks)
14
15    context = "\n\n".join(f"<doc id={i}>{c}</doc>" for i, c in enumerate(chunks))
16    if keep_oracle:
17        answer = f"##begin_quote##{quote}##end_quote##\nTherefore, {final}"
18    else:
19        answer = "The provided documents do not contain the answer."
20
21    return json.dumps({"messages": [
22        {"role": "system",
23         "content": "Answer only from the documents. Quote before you conclude."},
24        {"role": "user", "content": f"{context}\n\nQuestion: {question}"},
25        {"role": "assistant", "content": answer},
26    ]})
27
28
29with open("raft_train.jsonl", "w") as f:
30    for row in load_labelled_pairs():      # your own loader
31        f.write(raft_example(**row) + "\n")

The p_oracle fraction is the hyperparameter that matters: leaving the oracle out of some examples teaches the model to say the documents do not contain the answer instead of inventing one. The quote assertion gives you a groundedness rate you can score.

Forgetting: LoRA is not free

Every fine-tune trades general capability for specific capability; the question is how much. Biderman and colleagues (TMLR 2024) measured this across code and maths: LoRA learns less on the target domain than full fine-tuning and forgets less outside it, and full fine-tuning learns weight perturbations of rank ten to a hundred times higher than a typical LoRA configuration.

That is not a case for LoRA being free. Shuttleworth and colleagues (NeurIPS 2025) found LoRA introduces intruder dimensions, new high-ranking directions in the weight spectrum that full fine-tuning does not produce. Models carrying them score identically on your target evaluation while adapting worse to the next task. Same eval score, different model.

The defence is cheap and almost nobody builds it: 50 general-capability items unrelated to your task, run against base and tuned models on every training run, wired into CI so a regression fails the build. Two points on that suite is a reasonable threshold; schema validity should not drop at all.

01import json
02import pytest
03
04REGRESSION = [json.loads(line) for line in open("regression_50.jsonl")]
05MAX_ACCURACY_DROP = 0.02   # two points, absolute
06
07
08def score(generate, suite):
09    """generate(prompt) -> str. Returns (accuracy, schema_validity)."""
10    correct = valid = 0
11    for item in suite:
12        out = generate(item["prompt"])
13        valid += int(schema_ok(out, item.get("schema")))      # your validator
14        correct += int(grade(out, item["reference"]))         # your grader
15    return correct / len(suite), valid / len(suite)
16
17
18@pytest.fixture(scope="session")
19def scores(base_generate, tuned_generate):   # both defined in conftest.py
20    return score(base_generate, REGRESSION), score(tuned_generate, REGRESSION)
21
22
23def test_no_out_of_domain_regression(scores):
24    (base_acc, _), (tuned_acc, _) = scores
25    assert tuned_acc >= base_acc - MAX_ACCURACY_DROP, (
26        f"out-of-domain accuracy fell {base_acc:.3f} -> {tuned_acc:.3f}; "
27        "that is forgetting, not noise"
28    )
29
30
31def test_schema_validity_did_not_drop(scores):
32    (_, base_valid), (_, tuned_valid) = scores
33    assert tuned_valid >= base_valid, (
34        f"schema validity fell {base_valid:.3f} -> {tuned_valid:.3f}")

The fine-tuning vs RAG decision procedure

Here is the procedure in order. Answer honestly; the first question that matches is your answer, and you stop there.

1. Do you have a written eval set of at least 100 held-out items and a pass bar fixed in advance? If not, build it: nothing below is measurable without one.
2. Is the failure wrong or missing facts, and do those facts change, need citations, or need access control? Then it is RAG, and you stop here.
3. Are the facts static and small, and retrieval still misses? Try long context plus caching. If that fails, synthetic continued pretraining or prompt distillation is the legitimate knowledge fine-tune.
4. Is the failure wrong shape, tone, verbosity, tool-call format or refusal policy? Try a better prompt, five few-shot examples and caching. If not fixed, and you have 500 to 1,000 clean examples, LoRA SFT.
5. Is the failure cost or p99 latency, with quality already acceptable from a frontier model? Distil to a small open model: the highest-return fine-tune available in 2026.
6. Is the failure a verifiable score, such as unit tests or extraction F1? Then reinforcement fine-tuning or GRPO, where a reward function does the labelling.
7. Does retrieval work, but the model misreads chunks or falls for distractors? That is RAFT: tune it to read retrieval output and quote before answering.

As properties rather than a flow, the same trade-offs look like this.

The only row with a hard no in both freshness and provenance is fine-tuning alone, which is why it is the wrong default for anything an auditor might ask about. The format column matters most if you build systems that call tools, one of the few behaviours worth tuning on agentic AI builds.

How to decide empirically in one week

None of that tells you what is true for your data. This does, in a week.

• Day 1: build the eval set first. 100 to 300 real queries from production logs, graded, with a written pass bar. Never train on it.
• Day 1: build the regression suite too, 50 items of general capability. This is how you detect forgetting, and almost everyone skips it.
• Day 2: baseline with prompt, few-shot examples and caching. Record accuracy, p50 and p99 latency, and cost per thousand requests.
• Day 3: RAG, measuring retrieval recall@k separately from answer accuracy. Below roughly 85% recall@10, your problem is retrieval.
• Day 4: long context plus caching as an accuracy upper bound. Expect the token tax.
• Days 5 and 6: only now, LoRA SFT on 500 to 1,000 clean examples. Re-run the eval set and the regression suite.
• Day 7: decide on a table. Accuracy, retrieval recall, p99 latency, cost per thousand requests, regression delta, hours to update one fact.

The mistakes that invalidate the exercise are consistent. Evaluating on the training distribution. No held-out set. No regression suite. Comparing a tuned small model against an untuned frontier prompt. Counting training cost but not the dedicated endpoint, which at $5.49 to $8.00 an hour is $4,000 to $5,800 a month. Declaring victory on thirty examples.

Most of that is evaluation and observability work rather than training work, which is what our LLMOps practice spends its time on: eval harnesses, regression gates in CI, cost and latency tracing. For the compressed version, download the fine-tune decision checklist.

FAQ

Does fine-tuning add new knowledge to an LLM?

Barely, and unreliably. Ovadia and colleagues at Microsoft (EMNLP 2024) found fine-tuning on the documents containing the answers lifted Mistral-7B only from 0.481 to 0.504, while giving the same base model those documents through retrieval reached 0.875. One model tested got worse. Use retrieval for facts.

Is RAG cheaper than fine-tuning?

Usually, because the costs sit in different places. Training is cheap: roughly $10 to $16 of GPU time for a ten-thousand-example LoRA run on an 8B model. Serving a dedicated tuned endpoint costs roughly $4,000 to $6,000 a month, against $45 to $70 for a vector store.

Can you use RAG and fine-tuning together?

Yes, and the combination is the strongest result in the literature. RAFT fine-tunes a model to read retrieved chunks, including distractors, and to quote its source before answering. On PubMed it took LLaMA2-7B from 58.8 with plain RAG to 73.3. Fine-tune the reading; retrieve the facts.

How many examples do you need to fine-tune an LLM?

Plan on 500 to 1,000 clean, consistent examples for a behaviour or format change, and more if the task is varied. Below a few hundred you are mostly fitting noise and should stay with few-shot prompting. Label consistency matters far more than raw volume.

Does fine-tuning cause hallucinations?

It can, measurably. Gekhman and colleagues at Google (EMNLP 2024) showed that examples containing new knowledge are learned slowly, and as the model fits them its hallucination rate on facts it previously answered correctly rises linearly. Fine-tuning facts in trades a known gap for confident, unsourced errors.

Is LoRA as good as full fine-tuning?

Not identical, and the difference cuts both ways. Biderman and colleagues (TMLR 2024) found LoRA learns less on the target domain but forgets less outside it. Shuttleworth and colleagues (NeurIPS 2025) found LoRA adds intruder dimensions: the same eval scores, but worse sequential adaptation to later tasks.

Does a 1M-token context window make RAG obsolete?

No, it turns the question into a cost decision. The Token Tax paper (June 2026) measured long context at 73.1% accuracy against 65.4% for semantic RAG, at roughly 26 times the per-query token cost. Retrieval also gives citations, deletion and access control.

What replaces OpenAI fine-tuning?

OpenAI stops accepting new fine-tuning jobs on 6 January 2027, with earlier cut-offs already in force for new and inactive organisations. The alternatives are Amazon Bedrock, including reinforcement fine-tuning, Vertex AI for Gemini models, managed LoRA on Together or Fireworks, and self-hosted training on open weights.

Bring the eval set, not the fine-tune

Zenthos runs this procedure with clients, usually in the week described above, and most of the time the answer is not a fine-tune. When it is, it is nearly always distillation for cost and latency: the fine-tuning work itself, the evaluation and observability that tells you whether it worked, and the AI and machine learning engineering around it.

If you are weighing this decision now, get in touch for a free consultation. Bring your eval set — and if you do not have one, that is the first thing we build together.