Fine-Tune, Prompt, or RAG? A Decision Guide

Fine-tuning is the most over-reached-for tool in the LLM toolbox. The instinct is understandable: the model is almost right, so surely a little training on your data will close the gap. Sometimes it will. Often the gap isn't the kind fine-tuning closes — and you find that out after weeks of work. This is a guide to deciding before you spend the weeks.

First, what each technique actually changes

The three options people lump together solve different problems. Conflating them is where most wasted effort begins.

• Prompting shapes behavior at inference time using the model's existing capabilities. Cheapest to change, instant to iterate, nothing to maintain.
• RAG (retrieval-augmented generation) injects knowledge at inference time by fetching relevant documents and putting them in the context. The model doesn't learn the facts — it reads them, fresh, every call.
• Fine-tuning changes the model's weights to bias its behavior: format, tone, structure, task-specific patterns. What it is not reliably good at is teaching new facts.

That last point is the one teams learn the expensive way. In a controlled comparison of knowledge injection, Ovadia et al. (2023) found that RAG consistently outperformed unsupervised fine-tuning for getting facts into a model — both for knowledge seen during training and for genuinely new knowledge — and that RAG alone often beat RAG combined with fine-tuning. If your problem is "the model doesn't know X," fine-tuning is usually the wrong lever.

A decision order that saves weeks

Work down this list. Stop at the first step that meets your bar. Each step is an order of magnitude cheaper to change than the next.

1. Prompt engineering and few-shot

Start here, always. A clear instruction, a couple of worked examples, and a defined output schema close a surprising share of gaps. It costs an afternoon and zero infrastructure. If structured output is the problem, constrained decoding or a JSON schema usually solves it without touching weights.

2. RAG, if the gap is knowledge

If the model needs facts it doesn't have — your docs, your catalog, anything that changes — reach for retrieval before training. RAG keeps knowledge current without a retrain, lets you cite sources, and fails gracefully: a bad retrieval is visible and fixable, a fact baked wrongly into weights is not.

3. Fine-tune, if the gap is behavior

Fine-tuning earns its cost when the problem is consistency of form, not supply of facts:

• You need a specific tone, style, or output structure that prompting can't reliably enforce
• A specialized task where a smaller fine-tuned model matches a larger general one at a fraction of the inference cost
• Latency or cost pressure: shorter prompts (because the behavior is now in the weights) and a smaller model
• Privacy or compliance constraints that rule out a third-party API and push you to a model you host

Note that the strongest case for fine-tuning is usually economic, not capability: distilling a known behavior into a cheaper, faster, self-hosted model — not unlocking something the model couldn't otherwise do.

If you do fine-tune: what actually matters

Data quality dominates quantity

The most counterintuitive result in the alignment literature is how few examples it takes when they are good. In LIMA (Zhou et al., 2023), a model fine-tuned on just 1,000 carefully curated examples produced strikingly strong results — evidence that fine-tuning mostly surfaces capability the base model already has, and that a small, clean, consistent dataset beats a large noisy one. Practically:

• Consistency: every example follows the same format and voice you want at inference
• Coverage: include the edge cases and failure modes, not just the happy path
• Correctness: a wrong example actively teaches the wrong thing — verify them
• Deduplication: near-duplicates inflate your count and invite overfitting

Use parameter-efficient methods

Full fine-tuning is rarely necessary. LoRA (Hu et al., 2021) freezes the base weights and trains small low-rank matrices instead, cutting trainable parameters by orders of magnitude while staying close to full fine-tuning quality. QLoRA (Dettmers et al., 2023) adds 4-bit quantization, making it possible to fine-tune very large models on a single GPU. The adapters are tiny (megabytes, not gigabytes), so you can keep several task-specific ones and swap them at serve time.

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                 # rank: higher = more capacity, more params
    lora_alpha=32,        # scaling
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()
# trainable params: ~0.1% of the full model

Evaluate on behavior, not perplexity

Perplexity and BLEU correlate poorly with whether the model does the job. Before you train, define how you will know it worked: task-specific metrics (exact match for structured output, F1 for extraction), a small human-rated set for tone and correctness, and an A/B comparison against your current prompt-only baseline. If you can't measure the improvement, you can't claim it.

What fine-tuning won't fix

It won't make a model know facts it was never reliably taught — that is what retrieval is for. It won't paper over a vague task definition; if you can't write the instruction clearly, you can't build the dataset clearly either. And it isn't a one-time act: a fine-tuned model drifts as your data and base models move, so budget for periodic retraining or don't start.

The short version

Sources

• Ovadia et al., "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs" (2023) — arXiv:2312.05934
• Zhou et al., "LIMA: Less Is More for Alignment" (2023) — arXiv:2305.11206
• Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021) — arXiv:2106.09685
• Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" (2023) — arXiv:2305.14314