AI & Machine Learning for Developers
Retrieval-Augmented Generation (RAG)
Retrieval-augmented generation, usually shortened to RAG, is a pattern for answering questions with a language model by first retrieving relevant documents — usually with the embeddings and vector search covered elsewhere in this section — and then giving the model that retrieved text alongside the question, so it can ground its answer in specific source material rather than only what it learned during training. It's one of the most common ways developers connect a general-purpose language model to an organization's own, current, or private data.
Why it matters
- It addresses a real, specific limitation
- A language model's training data has a cutoff and doesn't include an organization's private documents; RAG lets the model answer using content it was never trained on, by handing that content to it at question time.
- It reduces, but does not eliminate, hallucination
- Grounding an answer in retrieved text makes fabrication less likely and easier to catch, since the source can be checked, but the model can still misread or misrepresent what it was given.
- It's cheaper and faster to update than retraining a model
- Adding, removing, or correcting a document in the retrieval source takes effect on the next query; retraining or fine-tuning a model to "know" the same fact is a much larger undertaking.
- Answer quality depends on retrieval quality, not just the model
- If the retrieval step returns the wrong or irrelevant documents, a strong language model still produces a poor answer — the two halves of the system are equally load-bearing.
The pattern: retrieve, then generate
A RAG system runs in two steps for each question. First, retrieval: the user's question is turned into an embedding (see this section's embeddings guide) and used to search a store of documents — often a vector database — for the passages most relevant to the question. Second, generation: those retrieved passages are inserted into the prompt sent to the language model, along with the original question and an instruction to answer using the provided material. The model then generates an answer grounded in that specific text, rather than answering purely from what it learned during training. Which passages get retrieved, how many, and how they're formatted into the prompt are all real design decisions that directly affect answer quality.
def answer_question(question, document_store, llm):
# 1. Retrieve: find passages relevant to the question
query_vector = embed(question)
relevant_passages = document_store.search(query_vector, top_k=5)
# 2. Augment: build a prompt that includes the retrieved text
context = "\n\n".join(relevant_passages)
prompt = (
"Answer the question using only the context below.\n\n"
"Context:\n" + context + "\n\nQuestion: " + question
)
# 3. Generate: the model answers grounded in that context
return llm.generate(prompt)Where RAG breaks down
RAG is not a fix for every limitation of a language model, and it introduces failure modes of its own. If the retrieval step misses the genuinely relevant document — because of a poor embedding match, a query worded differently from the source text, or a document that was never indexed — the model has no way to know that and will either answer from its own training data anyway or produce a plausible-sounding but ungrounded answer. Even with the right passages retrieved, the model can still misread, combine, or overstate what they say, especially when a question requires reasoning across several retrieved pieces rather than restating one. And the retrieved text is only as accurate, current, or unbiased as the source documents it was pulled from — RAG can confidently ground an answer in a source that is itself wrong.
Mistakes people make here
- Assuming RAG eliminates hallucination
- RAG reduces one specific cause of fabrication (the model having no relevant information at all) but doesn't prevent the model from misreading or embellishing the text it was actually given.
- Not checking what was actually retrieved
- When an answer is wrong, the fastest diagnosis is usually to look at which passages retrieval returned — the failure is very often on the retrieval side, not the generation side, and skipping this check leads to tuning the wrong part of the system.
- Retrieving too little or too much context
- Too few passages and the genuinely relevant information may be missing; too many and the important passage can get diluted among irrelevant ones, or exceed what the model can be expected to weigh evenly.
- Treating document chunking as an afterthought
- How source documents are split into retrievable pieces has a large effect on retrieval quality — a chunk that cuts a relevant sentence in half, or that mixes two unrelated topics together, will retrieve and read worse regardless of how good the embedding model is.
Strengths and trade-offs
Where it is strong
- Lets a language model answer using current, private, or organization-specific data without retraining the model itself.
- Answers can be made checkable, by showing which source passages they were grounded in.
- Updating the knowledge a system can draw on is as simple as updating the document store, not retraining anything.
The trade-offs
- Still allows hallucination — grounding reduces but does not remove the risk of a model misreading or embellishing what it was given.
- Answer quality is capped by retrieval quality; a strong model fed the wrong documents still gives a wrong or irrelevant answer.
- Adds real infrastructure and design surface area: an embedding step, a vector store, a chunking strategy, and a prompt template all now need to be built and maintained.
- Retrieved sources can themselves be outdated, biased, or wrong, and RAG has no built-in way to detect that.
Who needs this
Developers building a question-answering or search-style feature on top of a language model, especially over private or frequently changing documents, need this pattern. If your use case doesn't involve any external documents — pure conversation or generation from the model's own knowledge — RAG isn't the relevant tool.
Questions about retrieval-augmented generation (rag)
- Is RAG the same as fine-tuning a model?
- No. Fine-tuning changes the model's own parameters using additional training; RAG leaves the model unchanged and instead supplies relevant text at the moment of answering. They solve overlapping but different problems and can be combined.
- Does RAG guarantee accurate answers?
- No. It grounds answers in retrieved text, which makes them more checkable and generally more accurate when retrieval works well, but neither retrieval nor generation is guaranteed to be correct.
- Do I need a vector database to build a RAG system?
- It's the common choice for anything beyond a small, fixed set of documents, because it makes the retrieval step fast at scale, but a small enough document set can be searched with simpler methods.
- Why not just put all the documents in the prompt directly?
- Language models have a limited amount of text they can accept in one request, and even within that limit, giving a model only the relevant passages tends to produce better, more focused answers than burying them in irrelevant material.