What is RAG? Retrieval-Augmented Generation Explained

The Problem RAG Solves

LLMs like GPT-4 and Claude are trained on data up to a certain cutoff date. They don't know about your company's internal documents, your product's latest features, or anything that happened after their training ended. They also hallucinate — confidently stating false information when they don't know the answer.

Retrieval-Augmented Generation (RAG) solves this by giving the LLM access to a knowledge base at query time. Instead of relying purely on memorized training data, the model retrieves relevant documents and uses them as context to generate accurate, grounded answers.

How RAG Works — Step by Step

RAG Pipeline Code Example

// Simplified RAG pipeline with OpenAI + Chroma
import { OpenAI } from 'openai';
import { ChromaClient } from 'chromadb';

const openai = new OpenAI();
const chroma = new ChromaClient();
const collection = await chroma.getCollection({ name: 'docs' });

async function ragQuery(userQuestion: string) {
  // 1. Embed the user's question
  const queryEmbedding = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: userQuestion,
  });

  // 2. Retrieve top 5 relevant chunks
  const results = await collection.query({
    queryEmbeddings: [queryEmbedding.data[0].embedding],
    nResults: 5,
  });

  const context = results.documents[0].join('\n\n');

  // 3. Generate answer with context
  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [
      {
        role: 'system',
        content: 'Answer based only on the provided context. If the answer is not in the context, say so.',
      },
      {
        role: 'user',
        content: `Context:\n${context}\n\nQuestion: ${userQuestion}`,
      },
    ],
  });

  return response.choices[0].message.content;
}

Chunking Strategies

How you split documents dramatically affects retrieval quality:

Vector Databases Compared

Advanced RAG Techniques