How to Build a RAG Pipeline with LlamaIndex in Python

Large language models are powerful, but they have a fundamental limitation: they only know what was in their training data. If you need an LLM to answer questions about your company docs, a research paper, or a codebase it has never seen, you need RAG — Retrieval-Augmented Generation.

RAG works by retrieving relevant document chunks from a knowledge base, then feeding them to the LLM as context. The model reads the context and generates a grounded answer. No fine-tuning required.

This tutorial walks through building a working RAG pipeline with LlamaIndex — from installing dependencies to querying your own documents. Every code block is tested with LlamaIndex 0.12+ and Python 3.12.

1. What Is RAG and Why Use It?

RAG combines two steps: retrieve relevant information from a document store, then generate an answer using that information as context. This approach solves three real problems:

• Knowledge cutoffs: The model answers based on your documents, not just its training data.
• Hallucinations: Grounding the model in retrieved chunks reduces fabricated answers.
• Private data: You can query internal wikis, PDFs, and codebases without exposing them to the model provider.

LlamaIndex is a Python framework purpose-built for this workflow. It handles document loading, chunking, embedding, indexing, retrieval, and response synthesis — so you focus on the logic, not the plumbing.

2. Prerequisites and Installation

You need Python 3.10+ and an OpenAI API key (or a local model — we cover that too). Start by creating a virtual environment and installing LlamaIndex:

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate   # On Windows: venv\Scripts\activate

# Install LlamaIndex core + OpenAI integration
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai

# Verify installation
python -c "import llama_index; print(llama_index.__version__)"

Set your OpenAI API key as an environment variable:

export OPENAI_API_KEY="sk-your-key-here"

3. Building a Basic RAG Pipeline

Let's build a minimal RAG pipeline that reads a text file and answers questions about it. This is the foundation — once it works, you can swap in different data sources and models.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

# 1. Configure the LLM and embedding model
Settings.llm = OpenAI(model="gpt-4o-mini", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

# 2. Load your documents
#    Create a folder called "data" and put a .txt or .pdf file inside
documents = SimpleDirectoryReader("data").load_data()
print(f"Loaded {len(documents)} document(s)")

# 3. Build the index (chunks + embeds all documents)
index = VectorStoreIndex.from_documents(documents)

# 4. Create a query engine and ask a question
query_engine = index.as_query_engine()
response = query_engine.query("What is this document about?")
print(response)

When you run this, LlamaIndex will:

1. Read all files in the data/ directory
2. Split each document into chunks (default: 1024 tokens)
3. Generate an embedding vector for each chunk using OpenAI's embedding model
4. Store the chunks and embeddings in an in-memory vector index
5. On query: embed the question, find the top-k most similar chunks, and pass them to the LLM as context

The LLM then reads the retrieved chunks and generates an answer grounded in your data.

4. Understanding Chunking and Node Parsing

Chunking determines how your documents are split. Too large, and the LLM gets irrelevant context. Too small, and you lose meaning. LlamaIndex uses a SentenceSplitter by default, but you can customize it:

from llama_index.core import Settings
from llama_index.core.node_parser import SentenceSplitter

# Custom chunk size: 512 tokens per chunk, 50 token overlap
Settings.node_parser = SentenceSplitter(
    chunk_size=512,
    chunk_overlap=50
)

# Now rebuild the index with the new chunking settings
index = VectorStoreIndex.from_documents(documents)

Key parameters:

• chunk_size — Maximum tokens per chunk. 256–512 works well for factual Q&A. 1024+ for longer narratives.
• chunk_overlap — Tokens shared between adjacent chunks. Prevents splitting a sentence across two chunks. 10–20% of chunk_size is a good default.

5. Tuning Retrieval for Better Answers

The default query engine retrieves the top 2 most similar chunks. For most use cases, you want more control:

from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.postprocessor import SimilarityPostprocessor

# Retrieve top 5 chunks
retriever = VectorIndexRetriever(
    index=index,
    similarity_top_k=5
)

# Filter out chunks with similarity score below 0.7
postprocessor = SimilarityPostprocessor(similarity_cutoff=0.7)

# Combine into a query engine
query_engine = RetrieverQueryEngine(
    retriever=retriever,
    node_postprocessors=[postprocessor]
)

response = query_engine.query("Explain the key concepts in detail")
print(response)

You can also inspect which chunks were retrieved — useful for debugging:

# See the source chunks used for the answer
for i, source_node in enumerate(response.source_nodes):
    print(f"--- Chunk {i+1} (score: {source_node.score:.3f}) ---")
    print(source_node.node.text[:200])
    print()

6. Streaming Responses for Better UX

For chat-like applications, stream the answer token by token instead of waiting for the full response:

# Enable streaming
query_engine = index.as_query_engine(streaming=True)

# Stream the response
response = query_engine.query("Summarize the document in 3 points")
response.print_response_stream()

# Or iterate manually
for token in response.response_gen:
    print(token, end="", flush=True)

7. Adding Chat Memory with ChatEngine

A query engine is stateless — each question is independent. For conversational RAG, use a ChatEngine that remembers previous turns:

from llama_index.core.chat_engine import CondensePlusContextChatEngine

chat_engine = index.as_chat_engine(
    chat_mode=CondensePlusContextChatEngine,
    llm=Settings.llm,
)

# First question
response = chat_engine.chat("What are the main topics covered?")
print(response)

# Follow-up — the engine remembers the context
response = chat_engine.chat("Tell me more about the second topic")
print(response)

# Reset conversation
chat_engine.reset()

The CondensePlusContextChatEngine works well for most cases: it condenses the chat history and the new question into a standalone query, then retrieves relevant context before answering.

8. Using Local Models Instead of OpenAI

If you want to run RAG entirely offline (or avoid API costs), use Ollama for both the LLM and embeddings:

# Install Ollama integration
pip install llama-index-llms-ollama llama-index-embeddings-ollama

from llama_index.core import Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import OllamaEmbedding

# Use a local model (requires Ollama running with the model pulled)
Settings.llm = Ollama(model="llama3.2", request_timeout=120)
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")

# Everything else stays the same
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What is this document about?")
print(response)

Make sure you have Ollama installed and the models pulled:

ollama pull llama3.2
ollama pull nomic-embed-text

9. Persisting and Loading the Index

Rebuilding the index from scratch on every run is wasteful. Persist it to disk and reload when needed:

# Save the index to disk
index.storage_context.persist(persist_dir="./storage")

# Load it later — no need to re-embed
from llama_index.core import StorageContext, load_index_from_storage
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

# Query as usual
query_engine = index.as_query_engine()
response = query_engine.query("What is this document about?")

10. Common Mistakes and Pitfalls

10.1. Using chunk_size that is too large

If your chunks are 2048+ tokens, the LLM receives too much context per chunk and retrieval precision drops. The model may answer based on an irrelevant paragraph that happens to be in the same chunk. Stick to 256–512 tokens for most use cases.

10.2. Forgetting to set temperature to 0 for Q&A

For factual retrieval, set temperature=0. Higher temperatures introduce randomness, which is good for creative writing but causes the model to paraphrase or add details not present in the source.

10.3. Not checking retrieved chunks

If your RAG system gives bad answers, the problem is usually retrieval, not generation. Always inspect response.source_nodes to see what chunks were retrieved. If the relevant information is not in the top-k chunks, no LLM can produce a correct answer.

10.4. Mixing embedding models

You cannot query an index built with text-embedding-3-small using text-embedding-3-large. The embedding model used for indexing must match the one used for querying. If you switch models, rebuild the index from scratch.

10.5. Ignoring token limits

If you retrieve too many chunks and the combined context exceeds the LLM's context window, the model will truncate or error. Use similarity_top_k responsibly — 3 to 5 chunks is a good starting point for most documents.

11. When to Use RAG vs Fine-Tuning

For most use cases involving "answer questions about my documents," RAG is the right choice. Fine-tuning is better when you want the model to adopt a specific writing style or reasoning pattern.