RAG or fine tuning: which one does your use case need?

Start with retrieval. That is the honest short answer for most projects, and the reason is a clean distinction: retrieval augmented generation (RAG) hands the model documents to read at the moment you ask a question, while fine tuning trains the model further so that it behaves differently on every question afterwards. Facts go in through retrieval. Behavior, format and tone go in through fine tuning. If your team's complaint is "it does not know our products", that is a retrieval problem. If the complaint is "it knows, but it never answers the way we need it to", that is a fine tuning problem. The two get confused constantly, and picking the wrong one costs months.

What retrieval actually does to your prompt

Retrieval is plumbing bolted onto a normal chat request. Your documents are split into chunks of roughly 200 to 800 tokens each. Every chunk is converted into an embedding, a list of a few hundred or few thousand numbers that places the text by meaning rather than by keyword, and stored in a vector database. When a question arrives, the question is embedded the same way, the closest chunks are pulled out, and they are pasted into the prompt above your question with an instruction along the lines of "answer using only the material below, and say so if it is not there".

The model is not consulting a database. It is reading text you put in front of it, exactly as if you had pasted it yourself. Three consequences follow from that, and they are the whole case for retrieval:

• It can quote and cite, because the source text is right there in the window.
• It updates instantly. Change the policy document, re-index it, and the next answer is current.
• Access control is possible. You can retrieve only the chunks a particular user is allowed to see.

The price is that the answer is only as good as the retrieval. If the right paragraph is not in the top few chunks, the model answers from general knowledge and sounds just as assured.

What fine tuning actually changes

Fine tuning takes a finished model and continues training it on your example pairs, nudging the weights so your patterns become more likely. It is the same machinery described in how an AI model is trained, just a short final stage on a small dataset instead of a months long run.

What that reliably teaches: output format, house tone, domain vocabulary, classification labels, when to refuse, how long an answer should be, and conventions nobody ever wrote down but every experienced agent follows. What it teaches badly: facts. A product detail that appears 40 times in your training file is competing against everything the model absorbed during pretraining, and it may surface in the wrong context or not at all. Push hard enough to make the facts stick and you get overfitting, where the model parrots training examples and loses general ability.

Two projects that went different ways

A staff handbook assistant. Four hundred pages of HR policy, revised every quarter, used by 600 employees who need to know whether a particular expense is claimable. This is retrieval, without argument. Staff need the clause quoted so they can check it, the document changes faster than any training cycle, and a wrong answer about parental leave is a real problem. Total build: chunk the handbook, index it, add a citation instruction.

A support bot with a house format. Every reply must open with a one line acknowledgment, give at most three numbered steps, include the ticket reference, never promise a refund, and close with the escalation line. You can describe all of that in a system prompt, and for a few hundred messages a day that is fine. At high volume, those instructions are resent as input tokens on every single call, and compliance still drifts on the awkward cases. Fine tuning on 500 approved past replies bakes the format in, shortens the prompt and makes the behavior consistent. Note that it did not teach the bot anything about the products, which still come from retrieval.

The comparison in one table

The cost line deserves a second look. Retrieval adds perhaps 1,000 to 4,000 tokens of context to every request, which is small per query and significant across a million of them. The arithmetic for that is in what tokens are and why they decide your bill. Fine tuning shifts the cost to the front: the training job itself is usually modest, but building and reviewing a clean dataset is real salaried time, and you repeat it when the base model is retired.

A decision path you can run this week

1. Write down 20 real questions people actually ask, including 5 awkward ones.
2. Paste the relevant source documents straight into a chat and ask all 20. No tooling.
3. Grade the answers. If they are good, your problem is retrieval plumbing at scale, not model capability, and you never needed fine tuning.
4. If the answers are factually right but badly shaped, spend a day on prompting and worked examples before anything else. Most format complaints die here.
5. If they are still badly shaped after that, and the volume justifies it, gather 300 to 500 corrected examples and fine tune.
6. If the answers are factually wrong or vague, build retrieval and measure whether the correct chunk reaches the prompt. That single measurement explains most RAG failures.

Before you commit to either

Check two things first. One, who owns what you build: if you fine tune on a provider's model, confirm in writing whether the resulting weights are portable, how long the base model is supported and whether your training data can be used to improve their systems. Those clauses belong in an AI vendor contract review. Two, decide how you will know it works, because both approaches fail quietly. Keep those 20 questions as a fixed test set and rerun them after every change. A system that cites its sources at least lets you check, which is one more reason retrieval is the safer place to start when made up answers would cost you something real.