Fine Tuning vs RAG Is the Wrong Question

The question arrives as a fork in the road. Should we fine-tune a model on our data, or build a retrieval pipeline over it? Budget gets allocated to one branch, a team spends a quarter on it, and the result underwhelms in a way nobody can quite diagnose.

The framing is the problem. Two separate studies dismantle it from different directions, and it is worth taking them in order.

Retrieval was never proposed as a replacement

Go back to the paper that introduced retrieval-augmented generation. Its architecture combines two memories: a parametric one, the pre-trained sequence-to-sequence model holding knowledge in its weights, and a non-parametric one, a dense vector index of Wikipedia reached through a neural retriever. The paper compares two formulations — one that uses the same retrieved passages across a whole generated sequence, and one that can draw on different passages for each token.

The results were strong enough to set the state of the art on three open-domain question answering tasks, beating both parametric sequence-to-sequence models and the task-specific retrieve-and-extract architectures of the day, and to produce more specific, diverse and factual language than a parametric-only baseline.

Notice what the design is not. It is not a system that avoids relying on model weights. It is a system that adds a second memory alongside them, which is what the word augmented in the name has been saying the whole time. The original framing was additive. The versus framing came later, from vendors and blog posts, not from the research.

Fine-tuning is bad at the specific job people hire it for

The second study asks the question directly: if you want a model to know something it did not know before, is unsupervised fine-tuning or retrieval the better way to put it there?

The finding is one-sided. Retrieval consistently outperformed unsupervised fine-tuning across knowledge-intensive tasks — not only for entirely new information, which you might expect, but also for knowledge the model had already encountered during training. The authors report that models struggle to absorb new factual information through unsupervised fine-tuning at all. The one mitigation they identify is exposure to many variations of the same fact during training, which is a real result and also a description of considerable work.

That is the finding to argue with if your plan is to fine-tune a model so it learns your product catalogue, your policy documents, or last quarter's numbers. You are choosing the weaker instrument for that particular job, and the study says so for facts the model has seen before as well as facts it has not.

The result that settles it

The third study is the one that makes the versus framing untenable, and it does so almost in passing.

Working through an agricultural domain, the authors built a pipeline that extracts information from PDFs, generates question and answer pairs from it, fine-tunes on those pairs, and uses GPT-4 to evaluate the outcome, testing across Llama2-13B, GPT-3.5 and GPT-4. Fine-tuning produced a gain of more than six percentage points. Retrieval produced about five. Applied together, the gains were cumulative, landing north of eleven points combined. A separate experiment on geographic knowledge saw answer similarity climb from 47% to 72% once the fine-tuned model could draw on information from other regions.

Substitutes do not add. If fine-tuning and retrieval were two routes to the same destination, stacking them would produce roughly the larger of the two numbers, not their sum. Getting the sum is evidence that they are fixing different defects.

One caveat, since the study's own method contains it: the evaluation stage is a model grading other models' answers, which carries a measurement offset that does not disappear because the numbers came from a paper. It is a reason to read the eleven points as a direction rather than a constant, and it is the same caution that applies to your own scores — see the three jobs LLM evaluation gets asked to do.

What fine-tuning is actually for

The clearest statement of this comes from the vendors, who have every commercial reason to sell you fine-tuning and describe it narrowly anyway.

OpenAI's guide lists the use cases: classification, nuanced translation, producing output in a specific format, correcting failures to follow instructions, generating chat messages with a particular tone and style, summarising with attention to the details you care about, image classification for vision models, and complex domain-specific reasoning tasks on reasoning models. The supported methods are supervised fine-tuning, vision fine-tuning, direct preference optimisation, and reinforcement fine-tuning. The guide also notes that prompt engineering may be all a given use case needs.

Read that list again and look for teaching the model new facts. It is not there. Every entry is about behaviour — shape, tone, format, adherence, consistency. That is what adjusting weights is good at, and it is a genuinely hard problem that prompting solves only partially and expensively, in tokens you pay for on every request.

The decision that replaces the fork

Ask what kind of failure you are looking at.

If the model produces answers that are factually wrong, out of date, or unattributable to a source, that is a knowledge failure and retrieval is the instrument. Fine-tuning on the same documents will produce a model that sounds more like your documents while remaining just as capable of inventing a price.

If the model has the right information available and does the wrong thing with it — ignores the format you asked for, adopts a tone the brand cannot ship, drifts off the instructions by the third turn, or misclassifies in a way no prompt seems to fix — that is a behaviour failure, and it is what fine-tuning is for.

Most production systems have both, which is exactly why the gains added up in the agriculture study. The sequence that usually works is to exhaust prompting first because it is reversible in an afternoon, add retrieval next because knowledge failures are the more common and more damaging kind, and fine-tune last, once you have a stable prompt and a working retriever and can therefore tell what the fine-tune actually changed.

Which requires being able to measure the difference at all. Running this sequence without an eval set means each step is a matter of opinion, and the strongest opinion in the room wins. Decide what you are measuring, and with which of the metrics you can actually supply data for, before you spend the quarter.