You have a corpus of knowledge the base model doesn't know about. Maybe it's product docs, customer history, internal policies, or a domain-specific knowledge base. Three families of approaches: retrieval-augmented generation (RAG), fine-tuning, and long context. Each has a sweet spot, and getting this choice wrong is the most common architectural mistake in LLM apps.
The three approaches at a glance
RAG (Retrieval-Augmented Generation)
Index your data into embeddings + a search engine. At query time, retrieve the most relevant chunks and stuff them into the prompt. The model reasons over the retrieved context.
Fine-tuning
Train (or LoRA-adapt) a model on your data. The knowledge becomes parameters. Modern fine-tuning is mostly used for style and format adaptation, not raw knowledge injection.
Long context
Use a model with a very large context window (1M+ tokens) and pass everything at once. Avoids the retrieval problem by skipping retrieval.
When RAG wins
- Your knowledge base changes frequently (you can re-index without retraining)
- You need source attribution ("this answer came from doc X")
- You need access control (different users see different documents)
- Your corpus is large enough that long context isn't cost-effective
- You want explainability — a human can inspect which docs informed the answer
RAG is the default for a reason: it scales linearly, costs are predictable, and you can iterate on retrieval quality without retraining anything.
When fine-tuning wins
- You need the model to learn a style, not just facts (tone, format, terminology)
- You need consistent low-latency behavior at scale
- You have a stable, well-labeled training set (at least 1K examples, often 10K+)
- Your task is narrow and well-defined
- You want to use a smaller, cheaper base model and lift its performance on your specific task
The mistake most teams make: trying to fine-tune for knowledge. Fine-tuning is bad at learning specific facts and great at learning patterns. If you want the model to know "our refund policy is 30 days," put it in the prompt. If you want the model to respond in your support team's voice, fine-tune.
When long context wins
- The data fits comfortably in context and isn't too dynamic
- The use case requires reasoning over the whole document, not selective retrieval
- Latency at the per-query level isn't critical (long context is slow — often 10-30s for full 1M-token reads)
- You want minimal infrastructure overhead (no vector DB to manage)
- Cost-per-query is acceptable for your use case (it's expensive)
Long context shines for legal review, financial analysis of full filings, and codebase-level reasoning — tasks where retrieving the "relevant" chunks would miss important context elsewhere in the document.
The honest answer: combine them
Production systems usually mix all three:
- Fine-tune a small model for the tone and format your product needs
- Use RAG to inject the dynamic knowledge it needs to be factually correct
- Reserve long context for the cases where you genuinely need full-document reasoning
The framework: figure out what part of the problem is style (fine-tune), what is facts (RAG), and what is holistic structure (long context). Then engineer accordingly.
A simple decision tree
- Does the task require facts that change? → RAG
- Does the task require a specific style/format? → Fine-tuning
- Does the task require reasoning over a whole document at once? → Long context
- Otherwise → start with RAG, it's the cheapest to iterate on
Common mistakes
- Reaching for fine-tuning first — it's the most expensive to iterate on. Always try prompt engineering + RAG before fine-tuning.
- Trying to RAG your way to style — no amount of retrieved context will make a model write in your voice consistently. That's a fine-tuning problem.
- Stuffing everything into long context — needle-in-haystack performance degrades past a few hundred thousand tokens. Even with 1M context windows, retrieval often produces better results.
- Not evaluating across approaches — assuming one is best for your use case. Build an eval set, run all three, compare.
Cost and latency tradeoffs (rough)
For a typical question-over-corpus query at modest scale:
- RAG: 1-3 seconds, $0.005-$0.02 per query, sub-linear with corpus size
- Long context (1M tokens): 10-30 seconds, $0.20-$2.00 per query, linear with input
- Fine-tuned small model: 0.5-2 seconds, $0.001-$0.005 per query, plus one-time training cost
These numbers shift constantly. The relative ordering — fine-tuning cheapest per-call, RAG middle, long-context expensive — has held steady.
Where to start
RAG. Always RAG first. It's the most learnable, most iterable, and most production-friendly of the three. Get good at it before you reach for the others. If you want a guided path through building production RAG, the JoinAI MasterClass dedicates two weeks to it specifically.
