You've asked ChatGPT about your own PDF collection and received a confident, polished answer that was completely wrong. Or perhaps you've tried to query an LLM about a contract you're reviewing, only to realize it has no idea what's actually in the document. These aren't edge cases—they're fundamental limitations of how large language models work.
Large language models are trained on public internet data up to a specific cutoff date. They don't know your emails, your research papers, your invoices, or your notes. And when they don't know something, they don't say "I don't know." Instead, they generate plausible-sounding text that may be entirely fabricated. This phenomenon is called hallucination, and it's not a bug that will be patched—it's inherent to how these models operate.
Retrieval-Augmented Generation (RAG) solves this problem by giving the LLM access to your documents at query time. Rather than asking the model to recall information from its training data, you first search your own document collection for relevant passages, then feed those passages to the model as context, and ask it to answer based on that context.
The term was coined in a 2020 paper by Lewis et al. at Facebook AI Research, and the architecture has since become the standard way to build question-answering systems over private data. A 2024 Anthropic study found that RAG systems reduced factual hallucinations by up to 94% compared to base LLMs on knowledge-intensive benchmarks.
Cloud-based RAG services exist—you upload your documents to OpenAI, Anthropic, or other providers, and they handle retrieval and generation. However, that means your data leaves your machine. The average cost of a data breach in 2024 was $4.88 million, according to IBM, and for lawyers, doctors, or researchers, sending client files or personal data to a third party may be outright prohibited.
Running RAG locally offers several advantages:
This guide walks through building a complete local RAG system step by step: document ingestion, chunking, embedding generation, vector storage, retrieval, and local generation. You'll also learn how to evaluate and optimize the system. No cloud APIs required.
RAG splits the problem into two parts. A retriever finds relevant information in your document collection, while a generator (the LLM) reads that information and produces a coherent answer. The key insight is that the LLM doesn't need to remember your data—it just needs to read it at query time.
A RAG system has three phases:
RAG addresses hallucination because the model is constrained to answer based on provided context. It gives you up-to-date knowledge because you control the document collection—add a new file and the system immediately knows about it. And with local deployment, your data never leaves your machine.
Fine-tuning modifies the model's weights by training it on your data. It's useful when you need the model to adopt a specific style, format, or behavior. However, fine-tuning doesn't give the model access to specific facts—it teaches patterns, not data. If you fine-tune a model on your documents and ask it a factual question, it may still hallucinate.
RAG is the right choice when you need the model to answer questions about specific documents with accuracy and verifiability. You can see exactly which chunks the model used to generate an answer. Fine-tuning is for behavior modification; RAG is for knowledge access. You can also combine both approaches, but for personal document systems, RAG alone is usually sufficient.
Key Takeaway: RAG doesn't teach the model your data—it hands the model the relevant data at query time. This is why it reduces hallucinations and stays current as you add documents.
The honest answer: it depends on what models you want to run. Here's a practical baseline:
For CPU-only systems, generation will be slow—perhaps 5-15 tokens per second with a 7B model. This is usable for occasional queries but frustrating for interactive chat.
You'll need:
A GPU dramatically speeds up both embedding generation and LLM inference. However, if you don't have one, you can still build a functional system. The bottleneck will be generation speed, not retrieval—vector searches are fast even on CPU.
Quantization reduces the precision of model weights (from 16-bit to 4-bit, for example) to shrink memory requirements. A 4-bit quantized Llama 3 8B model takes about 4.5 GB of storage and 6-8 GB of RAM, versus 16 GB for the full-precision version. The quality loss is modest—you might notice slightly less nuanced answers, but for most document querying tasks, it's an excellent trade-off.
Key Takeaway: You can get started with a 16 GB laptop and no GPU. Use quantized models and keep your collection under a few thousand documents.
Your ingestion pipeline needs to handle various formats. Python libraries make this straightforward:
If your PDFs are scanned images rather than text, you need Optical Character Recognition (OCR). Tesseract (via the pytesseract wrapper) is the standard open-source option. The critical caveat: OCR errors propagate into your RAG answers. A misread date or name will produce wrong answers, and you won't easily spot the error because the source text is hidden inside an image.
For high-stakes documents (contracts, medical records), consider commercial OCR or manual verification for key sections.
Raw extracted text is messy. You'll encounter:
Write cleaning functions to strip these artifacts. Normalize whitespace, fix common encoding issues, and remove boilerplate.
Store metadata alongside each chunk. At minimum:
Metadata enables filtering during retrieval—for example, "only search documents from 2023" or "only search contracts."
Key Takeaway: Garbage in, garbage out. The quality of your RAG system is bounded by the quality of your text extraction. Spend time on cleaning and OCR accuracy.
Your documents are too long to embed as single vectors—a 50-page PDF would produce one embedding that captures nothing specific. Conversely, you can't embed every sentence independently because the model loses context. Chunking is the art of splitting documents into retrievable pieces that are self-contained enough to be meaningful but specific enough to be findable.
The simplest approach: split text into chunks of N characters (or tokens) with an overlap of M characters between adjacent chunks. The overlap prevents relevant information from being split across a boundary and lost. A common starting point is 500-1000 characters with 50-100 characters of overlap.
Recursive chunking respects document structure. It tries splitting by paragraph, then by sentence if paragraphs are too long, then by phrase if sentences are too long. This produces more coherent chunks than fixed-size splitting.
Semantic chunking uses embedding similarity to find natural boundaries where the topic shifts. This is more computationally expensive but produces the most coherent chunks. LangChain and LlamaIndex both offer semantic chunking implementations.
There's no universal answer, but here's a guide:
The right size depends on your retrieval needs. If users ask narrow factual questions ("What was the budget for Q3?"), smaller chunks work better. If they ask for summaries of sections, larger chunks help.
Key Takeaway: Start with recursive chunking at 500-1000 characters with 10% overlap. Test with your real queries and adjust based on retrieval quality.
Embeddings are numerical vector representations of text that capture semantic meaning. "Dog" and "puppy" produce similar vectors; "dog" and "car" produce different ones. When you search, you convert your query to an embedding and find document chunks with the most similar vectors.
The embedding model determines the quality of your semantic search. A weak embedding model will miss relevant chunks; a strong one will surface them reliably.
For local use, your constraints are quality, speed, and memory:
The sentence-transformers library makes it easy to load and run these models locally.
With sentence-transformers, the code is straightforward:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(["Your text chunk here", "Another chunk"])
Embedding generation is CPU-bound but benefits from GPU acceleration. Process chunks in batches of 32-64 for efficiency. For a few thousand chunks, this takes minutes on CPU; for hundreds of thousands, you'll want a GPU.
Key Takeaway: Your embedding model is the foundation of retrieval quality. If you can afford the memory, BGE-base is a better choice than all-MiniLM for most document collections.
A vector database stores embeddings and supports fast similarity search. Three options dominate local use:
Chroma: The easiest to set up. It's a pure Python library that persists to disk. Chroma reports sub-100ms query latency for collections up to 1 million vectors on standard hardware. Ideal for personal projects.
FAISS: Facebook AI Research's library for efficient similarity search. It's not a database—it's a library—but it's extremely fast, handling billion-scale datasets in milliseconds on GPU. You manage persistence yourself.
Qdrant: A full-featured vector database with a server component. More complex to set up but offers advanced filtering, hybrid search, and production features.
For a personal document system:
Whatever you choose, you'll store:
The core operation is: given a query embedding, return the N most similar stored vectors. "Similarity" is typically cosine similarity or Euclidean distance. All vector DBs implement this efficiently.
# Chroma example
import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("my_documents")
collection.add(
embeddings=embeddings,
documents=chunks,
metadatas=metadatas,
ids=ids
)
results = collection.query(
query_embeddings=[query_embedding],
n_results=5
)
Key Takeaway: Start with Chroma. It's the least configuration and works fine for collections up to hundreds of thousands of chunks.
Basic retrieval is vector similarity: embed the query, find the closest chunks. This works well for conceptual queries ("What are the arguments for and against this policy?") but can miss exact matches.
Personal documents are full of names, acronyms, and exact strings. A semantic search for "What did Sarah say about the merger?" might miss a chunk that contains "Sarah" but uses different surrounding language. Hybrid search combines vector similarity with keyword search (BM25) and merges results.
This matters more than you might think. For documents with proper nouns and technical terms, hybrid search often produces substantially better retrieval than pure semantic search. Both LangChain and LlamaIndex support hybrid search with Chroma or Qdrant.
If you've stored metadata, you can filter before or after vector search. Examples:
This reduces noise and improves precision.
Vector search retrieves candidates based on embedding similarity. A cross-encoder model (like BGE-reranker) takes a query and a candidate chunk together and scores their relevance more accurately. It's slower than vector search, so you use it as a second stage: retrieve the top 20-50 chunks with vector search, then re-rank to the top 5-10 with the cross-encoder.
This two-stage approach substantially improves answer quality. For personal document systems, the added latency (a few hundred milliseconds) is usually acceptable.
Key Takeaway: Hybrid search plus re-ranking is the difference between a system that sometimes finds the right answer and one that reliably does.
Your choices for local generation models:
For document Q&A, any of these work. Start with Llama 3 8B if you have 16 GB RAM; otherwise, Phi-3 or Mistral 7B.
Ollama is the simplest option. It handles model downloads, quantization, and provides an OpenAI-compatible API. Install it, then:
ollama pull llama3
ollama run llama3
llama.cpp is more manual but gives you finer control over quantization and inference parameters. It's the right choice if you need to squeeze performance from limited hardware.
Models for local inference are distributed in GGUF format, a quantized format created by the llama.cpp project. Quantization levels are typically Q4_K_M (good balance) or Q5_K_M (higher quality, more memory). Ollama handles GGUF conversion automatically.
The prompt structure matters enormously. A basic template:
You are a helpful assistant. Answer the question based only on the provided context.
If the answer cannot be found in the context, say "I couldn't find this information in the provided documents."
Context:
{retrieved_chunks}
Question: {user_question}
Answer:
Key principles:
Local models have context windows of 4K-128K tokens. Llama 3 8B has 8K context. If your retrieved chunks exceed this, the model will truncate or fail. Solutions:
Key Takeaway: The prompt is where RAG succeeds or fails. Ground the model explicitly, and it will follow your documents. Leave it ungrounded, and it will hallucinate.
Frameworks abstract away the plumbing. Here's a minimal LangChain pipeline:
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.llms import Ollama
from langchain.chains import RetrievalQA
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader
# Load documents
loader = PyPDFLoader("my_document.pdf")
documents = loader.load()
# Split into chunks
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=800,
chunk_overlap=80
)
chunks = text_splitter.split_documents(documents)
# Create embeddings and vector store
embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory="./chroma_db"
)
# Set up the LLM
llm = Ollama(model="llama3")
# Build the RAG chain
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever(search_kwargs={"k": 5})
)
# Ask a question
answer = qa_chain.invoke("What are the key terms of this contract?")
This pipeline does the following:
Test with queries you know the answer to. Ask about specific facts, dates, and numbers in your documents. If the system gets these wrong, your retrieval or chunking needs adjustment.
| Symptom | Likely cause |
|---|---|
| Wrong answers | Retrieval is returning irrelevant chunks |
| "I couldn't find this" when the answer exists | Chunking split the answer across boundaries |
| Slow generation | Model too large for your hardware |
| Missing recent documents | Embeddings not updated after adding files |
Key Takeaway: A working RAG pipeline is ~30 lines of code with LangChain. The real work is in tuning retrieval quality, not in setup.
Recall measures: of all the chunks that contain the answer, how many did your system retrieve? Precision measures: of all chunks retrieved, how many were relevant?
Build a test set of 20-50 questions with known answers and known source chunks. Run retrieval on each question and compute recall@k (did the correct chunk appear in the top k results?). This gives you a quantitative baseline to improve against.
RAGAS is a framework that scores RAG outputs on faithfulness (is the answer grounded in the retrieved context?) and answer relevance (does it address the question?). It uses an LLM as a judge, which means it can run locally.
Manual review is more reliable for small sets. Read the answers, check the retrieved chunks, and identify failure patterns.
Systematic tuning:
Common wins:
RAG is not a set-and-forget system. As you add documents, retrieval quality shifts. Re-run your test set periodically to catch regressions.
Key Takeaway: You can't improve what you don't measure. Build a test set, track retrieval metrics, and iterate.
Verify that no data leaves your machine. Check that:
Several libraries (Hugging Face transformers, some vector DBs) send usage statistics by default. Check documentation and disable:
# Hugging Face
import huggingface_hub
huggingface_hub.constants.HF_HUB_DISABLE_TELEMETRY = True
Your vector database stores text chunks—potentially sensitive ones. Encrypt the database directory, or store the entire system on an encrypted volume. On macOS, use FileVault; on Linux, LUKS; on Windows, BitLocker.
Key Takeaway: "Local" doesn't automatically mean "private." Audit your tooling for telemetry and encrypt your storage.
A lawyer built a system with 500 case files (PDFs) using Llama 3 8B and ChromaDB. The system lets them find precedents, summarize rulings, and locate specific clauses—without sending confidential client data to cloud services. Metadata filtering by case type and date narrows searches.
A researcher indexed 200 papers on a laptop using FAISS and a 7B Mistral model. Semantic search across papers enables literature review drafts with citations, and hybrid search handles the author names and journal abbreviations that pure semantic search misses.
A small business owner scanned and OCR'd 1,000 invoices and contracts. Using hybrid search, they can ask "What was the total spend with vendor X in 2023?" and get answers with citations to specific invoices. The system flags anomalies by comparing retrieved amounts against expectations.
A medical student created a knowledge base from textbooks and lecture notes. The local RAG system generates quiz questions and study summaries. Because everything runs locally, personal notes about patients remain private.
A journalist archived 10 years of interview transcripts and articles. With re-ranking, the system fact-checks quotes and finds historical context. The ability to see which source chunks back each claim is essential for verification.
Building a local RAG system involves seven steps:
The trend is toward smaller, more capable models running on consumer hardware. Gartner predicted that by 2026, 60% of enterprises will have implemented RAG-based AI systems, up from less than 10% in 2023. As local models improve and quantization techniques advance, the quality gap between local and cloud RAG will continue to narrow.
Start small. Index a few dozen documents, ask questions, and examine the retrieved chunks. Build a test set and measure retrieval quality. Then iterate—try different chunk sizes, embedding models, and search strategies.
The tools you need are free and open source. The hardware you need is probably on your desk already.
A minimum of 8 GB RAM is needed for small models (3-4B parameters). For comfortable use with 7-8B models, 16 GB RAM is recommended. A GPU with 8+ GB VRAM will substantially speed up generation, but CPU-only systems are workable for occasional queries.
Start with all-MiniLM-L6-v2 for simplicity. If retrieval quality is insufficient, try BGE-base or nomic-embed-text. Evaluate with a test set of your own queries—benchmark quality matters less than performance on your specific documents.
For standard documents, 500-1000 characters with 10% overlap is a good starting point. Smaller chunks (200-500) work better for narrow factual queries; larger chunks (1000-2000) help with summarization tasks. Test and adjust.
Yes, but you need OCR (Tesseract is the standard open-source option). Be aware that OCR errors will propagate into your answers. For high-stakes documents, verify OCR quality or use commercial OCR services on non-sensitive content.
Add new documents by running them through the ingestion pipeline and adding their chunks to the vector database. There's no retraining involved—the LLM stays the same; you're just expanding the searchable index.
Most likely retrieval failure: the relevant chunk isn't being found, or irrelevant chunks are being retrieved. Check what the retriever returns for your query. Common fixes: switch to hybrid search, add a re-ranker, adjust chunk size, or improve metadata filtering.
RAG retrieves relevant documents at query time and gives them to the model as context. Fine-tuning modifies the model's weights to change its behavior. RAG is for knowledge access and verifiability; fine-tuning is for style and format. Use RAG for document Q&A.
It can be, but not automatically. You must ensure all models run locally, disable telemetry in your libraries, and encrypt your storage. If you use cloud OCR or API-based models at any point, your data leaves your machine.
Build a test set of 20-50 questions with known answers. Measure retrieval quality (did the correct chunk appear in the top results?) and answer quality (was the final answer correct and grounded?). Use RAGAS for automated scoring or review manually.
Ready to take control of your data? Start building your own local RAG system today and explore the power of private, on-device AI. Share your experiences and questions in the comments below!