RAG vs fine-tuning: choosing how a model learns your domain

The question is usually framed as a choice between two ways of teaching a model about a business. That framing hides the useful distinction. Retrieval-augmented generation supplies knowledge: it fetches relevant text at request time and places it in the context window. Fine-tuning adjusts behaviour: it changes the model's weights so that it writes in a certain format, follows a certain policy or performs a narrow task more reliably.

Our default is retrieval for anything that is a fact, and fine-tuning only when there is evidence that prompting and retrieval cannot produce the required behaviour. The rest of this article explains why, and where that default is wrong.

With retrieval, the model is unchanged. A query is turned into a search over an index, the best passages are selected, and the model answers using them. Knowledge lives in a datastore that can be updated, filtered and audited.

With fine-tuning, a base model is trained further on example input and output pairs, often through low-rank adapters such as LoRA so that only a small set of additional weights is learned. Knowledge and behaviour are compressed into parameters. Nothing can be looked up, cited or deleted individually afterwards.

Business knowledge changes, belongs to different audiences and sometimes has to be removed. Those three properties map directly onto what a retrieval system does well and a tuned model does badly.

Retrieval moves the difficulty into search, and search is where these systems usually fail. If the right passage is not retrieved, the model either answers from its general training or confidently fills the gap. Fixed-size chunking splits tables and separates a clause from the heading that gives it meaning. Dense embeddings alone miss exact identifiers such as product codes and clause numbers, which is why we combine vector search with keyword search such as BM25 and rerank the merged candidates.

Retrieval also costs tokens and latency on every request, and it struggles with questions that need a view of the whole corpus, such as a summary of every contract with a given term. Those need aggregation pipelines or structured queries, not a larger top-k.

Fine-tuning is the right tool when the problem is how the model responds, not what it knows. Typical cases are a strict output structure that prompting cannot hold at volume, a domain register such as clinical or legal drafting style, a classification or extraction task with thousands of labelled examples, and a requirement to run a small model cheaply or on private infrastructure. In that last case, tuning a small model on the outputs of a larger one can bring a narrow task within a latency and cost budget that the large model cannot meet.

It is also worth considering when the prompt has grown into pages of instructions and examples that are paid for on every call. Moving stable behaviour into weights shortens the prompt.

A fine-tuned model is a build artefact with its own lifecycle. It needs a curated training set, a held-out evaluation set, a record of which data produced which version, and a plan for the day the base model is retired and the tuning has to be repeated on its successor. Training data that contains personal or confidential information is now embedded in an artefact that is hard to inspect.

There are quality risks too. Narrow tuning can degrade general ability, the effect known as catastrophic forgetting, and a model tuned on facts will state stale facts with full confidence after the facts change. This is the main reason we do not use fine-tuning as a knowledge store.

We work through the options in order of cost and reversibility, and stop at the first one that meets the evaluation target.

Fine-tuning first is reasonable when the task is closed and the knowledge is static: classifying support tickets into a fixed taxonomy, extracting fields from a known document type, or running offline on a device with no index to query. Retrieval adds moving parts that such a task does not need.

Either way, the decision should be made with measurements. An evaluation set that separates retrieval failures from generation failures shows which technique addresses the errors actually observed, and it prevents an expensive training run aimed at a problem that better search would have fixed.