Local RAG Over Your Own Docs: A Working Setup
In short
Local RAG means embeddings, vector store, and generation all run on your machine, so the documents never leave it. The minimal stack is three pieces: Ollama serving an embedding model and a chat model, a vector store, and a retrieval script that stitches them together. But the first question is whether you need retrieval at all. Anthropic's own guidance is blunt: if your knowledge base is under 200,000 tokens — roughly 500 pages — put the whole thing in the prompt and skip RAG entirely. Most "chat with your documents" projects are below that line. Below it, RAG adds a chunking bug surface and an index to keep fresh in exchange for nothing. Above it, the parts that actually decide answer quality are chunk boundaries and hybrid search, not which vector database you picked. Build an eval set of twenty real questions before you build the pipeline, or you will have no way to tell whether any of it works.
When RAG beats just pasting the file
Start by disqualifying yourself. Anthropic's contextual retrieval writeup states that for knowledge bases smaller than 200,000 tokens — about 500 pages — you can include the whole thing in the prompt with no need for RAG. Long-context models plus prompt caching made the simple path cheaper and faster than the clever one for small corpora.
So RAG earns its complexity when at least one of these is true:
- The corpus is genuinely large. Thousands of pages, a decade of tickets, a full wiki.
- It changes constantly. You cannot re-paste a moving target, and cache invalidation on a 200K-token prompt is its own problem.
- Answers must cite. Retrieval gives you a chunk with a source path. Stuffing the context window gives you a plausible sentence with no provenance.
- Locality matters. Your questions hit 3 of 4,000 documents and you want the model looking at those 3.
If none of those apply, cat docs/*.md | claude -p "..." is the correct architecture and you are done. Being honest about this is the difference between shipping something and maintaining a hobby.
The minimal local stack
Three components, and the interesting choice is only the middle one.
Generation and embeddings: Ollama. One binary, one ollama pull, an HTTP API on localhost. nomic-embed-text is the usual starting embedding model — 274 MB on disk with a 2K context window per its Ollama library page, which is small enough to run alongside a chat model on a laptop. Our Ollama setup card covers installation and model choice.
Vector store:
| Store | Runs as | Good for | Watch out for |
|---|---|---|---|
| sqlite-vec | SQLite extension, pure C | Single-machine, single-file index, embed anywhere including WASM | Explicitly pre-v1 — "expect breaking changes" per its README |
| LanceDB | Embedded library (Python/TS/Rust) | Larger corpora, columnar Lance format, multimodal | Heavier dependency; more concepts to learn |
| Chroma | Embedded or client/server | Fast prototyping, batteries included | Defaults hide decisions you should be making |
| Postgres + pgvector | Existing database | You already run Postgres and want one system | Operational overhead if you don't |
If you have never built this before, start with sqlite-vec or Chroma and accept that you will rewrite the storage layer once. Do not start with a distributed vector database. The retrieval quality problem is not a database problem.
The glue. Fifty to two hundred lines: walk the docs directory, chunk, embed each chunk via Ollama's embed endpoint, store vector plus source path plus chunk text, and at query time embed the question, pull top-k, build a prompt. Frameworks will do this for you and will also hide the two parameters that matter.
Chunking that does not ruin answers
This is where local RAG setups fail, and it is almost never the model's fault.
Split on structure, not character count. A markdown document has headings; a code file has functions; a PDF has sections. Fixed-size windows cut mid-sentence and mid-table, and a chunk that starts halfway through a numbered list is unusable no matter how good the embedding is. Split on ## boundaries first, then subdivide anything still too long.
Keep the chunk under the embedding model's context. nomic-embed-text handles 2K tokens. Feed it more and the tail is silently dropped — you get an embedding for the first part of your chunk and retrieve text the vector never saw. This is the single most common invisible bug in homemade pipelines.
Add context to each chunk before embedding. A chunk reading "This parameter defaults to 30 seconds" is meaningless standalone. Prefixing it with the document title and heading path — "billing/webhooks.md > Retry policy" — is a two-line change. Anthropic's measured version of this idea reduced top-20-chunk retrieval failure rate from 5.7% to 3.7%, a 35% relative reduction; adding BM25 keyword search on top brought it to 2.9%.
Run keyword search alongside vectors. Exact identifiers — error codes, function names, SKUs — are exactly what embeddings blur together and what SQLite's FTS5 or a plain rg nails. Union the two result sets. This is the highest-value fifty lines in the whole pipeline.
Store the source path with every chunk. Non-negotiable. Without it you cannot cite, and without citations you cannot audit whether the answer came from your documents or from the model's priors.
Measuring whether it actually helps
Almost nobody does this, which is why almost nobody knows whether their RAG works.
Write twenty to fifty real questions — the ones you actually asked last month — and for each, record which document contains the answer. That file is your eval set. Then measure one thing first:
Recall@k: for what fraction of questions does the correct document appear in the top k retrieved chunks? If recall@10 is 60%, no prompt engineering will save you; the model never sees the answer in four out of ten cases. Fix retrieval before touching the prompt.
Only after recall is acceptable should you look at answer quality, and there the cheapest honest method is reading twenty answers yourself. LLM-as-judge scoring is convenient and correlates loosely with what you care about; treat its scores as a regression alarm, not a measurement.
Re-run the eval every time you change chunk size, embedding model, or k. Without it you are tuning by vibes, and retrieval tuning by vibes has a reliable outcome: you convince yourself it improved.
Cost and privacy trade-offs
The honest version of the local-vs-cloud comparison:
Cost is usually not the reason. Cloud embedding APIs are priced per million tokens and embedding a personal corpus once is genuinely cheap — check the current rate on the provider's pricing page rather than trusting a number in a blog post. Local costs you electricity, disk, and your time. If you are going local purely to save money on a few thousand documents, the arithmetic probably does not support it.
Privacy is the reason. Client contracts under NDA, medical or HR records, unreleased code, anything covered by a data processing agreement you would rather not extend. With Ollama and a local vector store, the text never crosses a network boundary. That is a categorical guarantee, not a policy promise, and it is the only argument for local RAG that does not weaken under scrutiny.
Local models are weaker at synthesis. An 8B model quantized to 4 bits runs on consumer hardware and will summarize a retrieved chunk competently. Ask it to reconcile four contradictory documents and the gap against a frontier model shows immediately. A common middle path: retrieve locally, send only the retrieved chunks to a cloud model. Your index stays private; a few paragraphs per query do not. Decide deliberately whether that is acceptable — for NDA material it usually is not.
You own the maintenance. Re-indexing on document change, model upgrades that change embedding dimensions and invalidate the whole index, disk growth. No vendor handles this. Budget for it, or the setup quietly goes stale and starts answering from documents you deleted six months ago.
More configuration detail lives in our local RAG card, and the rest of the practical guides are in the docs section.
FAQ
Do I need RAG to chat with my own documents?
Often not. If everything fits under roughly 200,000 tokens — about 500 pages — Anthropic recommends putting the whole knowledge base in the prompt instead. RAG starts paying off with large corpora, frequently changing sources, or a hard requirement to cite which file an answer came from.
What is the simplest local RAG stack that actually works?
Ollama for both the embedding model and the chat model, nomic-embed-text for embeddings, sqlite-vec or Chroma for storage, and a short script that chunks on headings, embeds, and retrieves top-k. Everything runs on localhost. The script is a couple of hundred lines and you will understand all of it, which matters when retrieval goes wrong.
How large should my chunks be?
Small enough to fit inside the embedding model's context window — 2K tokens for nomic-embed-text — and aligned to document structure rather than a fixed character count. Split on headings first, subdivide what is still too long, and prefix each chunk with its title and heading path before embedding.
Is local RAG actually private?
If embeddings, vector store, and the generating model all run on your machine, yes: no document text leaves the host. The common leak is the hybrid setup where retrieval is local but the final answer is generated by a cloud API — that sends your retrieved chunks off-machine. Check which of the two you are actually running.
How do I know whether my RAG setup is any good?
Build an eval set of twenty to fifty real questions with the correct source document recorded for each, then measure recall@k — how often the right document lands in the top k results. Fix retrieval before touching prompts, and re-run the eval after every change to chunking, embedding model, or k.