重要なポイント
- RAG = upload documents + retrieval + local LLM answering. No training required.
- Five steps: (1) Load documents, (2) chunk into 500–1000 token pieces, (3) generate embeddings, (4) store in vector DB, (5) retrieve on query.
- Best embedding model: nomic-embed-text (137M, runs locally, 768-dim vectors).
- Best vector DB: Chroma (simple, embedded) for <1M documents; Qdrant (distributed) for production.
- As of April 2026, local RAG is faster and cheaper than cloud APIs. Quality depends on retrieval accuracy and prompt engineering.
How Does RAG Work Step-by-Step?
- 1Document ingestion: Load PDFs, text files, or web pages.
- 2Chunking: Split documents into 500–1000 token chunks (overlap 20% to prevent context breaks).
- 3Embedding: Convert each chunk to a vector (768–1536 dimensions) using a local embedding model.
- 4Storage: Store vectors in a vector database (Chroma, Qdrant, Milvus) with metadata (document name, page, timestamp).
- 5Query time: Convert user question to embedding, search vector DB for top K similar chunks (k=5–10).
- 6Context assembly: Combine retrieved chunks into a prompt with instructions for the local LLM.
- 7Generation: Local LLM generates answer based on retrieved context.
- 8Attribution: Return which documents the answer came from.
What Is the Optimal Chunking Strategy?
Chunking strategy determines retrieval quality. Bad chunking = relevant information split across chunks, retrieval fails.
Semantic chunking (recommended): Split by sentences or paragraphs, preserving meaning. Example: each paragraph = 1 chunk.
Fixed-size chunking: 500 tokens per chunk, 20% overlap. Simple but may split sentences.
Recursive chunking: Split by paragraphs first, then by sentences if too large. Preserves hierarchy.
As of April 2026, semantic chunking with 500–1000 token chunks and 20% overlap is optimal for most use cases.
# Python: semantic chunking example
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200, # 20% overlap
separators=["\n\n", "\n", ".", " "] # Split on paragraph, then sentence
)
chunks = splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")Which Vector Database Should You Use?
| Database | Type | Capacity | Setup Effort | Best For |
|---|---|---|---|---|
| Chroma | Embedded | <1M docs | — | Prototyping, small RAG |
| Qdrant | Distributed | Unlimited | — | Production, scalable |
| Milvus | Distributed | Unlimited | — | Enterprise, massive scale |
| Weaviate | Graph + Vector | Unlimited | — | Complex queries, relationships |
| Pinecone (cloud) | Managed | Unlimited | — | Serverless, hands-off |
What Embedding Model Should You Choose?
| Model | Vector Dimensions | Speed | Quality | Recommendation |
|---|---|---|---|---|
| nomic-embed-text (local) | — | Fast | Excellent | Best for local RAG |
| bge-m3 (local) | — | Fast | Excellent | Multilingual support |
| OpenAI text-embedding-3 (cloud) | — | Very fast | Best in class | Hybrid approach |
| Cohere (cloud) | — | Fast | Excellent | Production cloud RAG |
How Do You Optimize Retrieval Quality?
Retrieval quality determines RAG success. Good retrieval = good answers. Bad retrieval = hallucinations.
- Top K selection: Retrieve k=5–10 chunks. Higher k = more context (slower), lower k = fewer distractions.
- Similarity threshold: Filter results by minimum similarity score (e.g., >0.75). Avoids low-relevance chunks.
- Reranking: Use a reranker (cross-encoder) to re-rank retrieved chunks by relevance. Small accuracy boost.
- Hybrid search: Combine semantic search (embeddings) with BM25 keyword search. Catches documents with exact keywords.
- Query expansion: Expand user query with synonyms or related terms. Improves recall.
How Do You Evaluate RAG Quality?
RAG quality has two dimensions: (1) retrieval quality (did we get relevant chunks?), and (2) generation quality (did the LLM answer well?).
Retrieval evaluation: Create test queries with known correct documents. Measure precision (how many retrieved are relevant?) and recall (did we get all relevant documents?).
Generation evaluation: Run LLM on retrieved chunks, manually score answers (0–5 scale) for accuracy and completeness.
As of April 2026, automated evaluation tools (like Ragas) can measure retrieval and generation metrics automatically.
Production RAG Patterns
For production services, use these patterns:
- Caching: Cache embeddings of frequently-queried documents to avoid recomputing.
- Incremental indexing: Add new documents without reindexing everything. Qdrant and Milvus support this.
- Monitoring: Track retrieval latency, cache hit rate, and user feedback on answer quality.
- Fallback: If retrieval fails (no relevant chunks), respond with "I don't have information about that" instead of hallucinating.
- Versioning: Keep document versions for audit trails. Store which version was used for each answer.
Common Mistakes in Local RAG Implementation
- Chunking documents wrong. Too many small chunks = retrieval noise. Too few large chunks = split information. Test chunk sizes empirically.
- Not evaluating retrieval. Building RAG without testing if retrieval works is like building a car without testing the engine. Always measure precision/recall.
- Using generic embeddings for domain-specific documents. Legal, medical, or technical documents may need fine-tuned embeddings. Consider domain-specific models.
- Forgetting about update frequency. If documents change weekly, your vector DB gets stale. Build a pipeline to re-embed and update.
- Expecting RAG to replace fine-tuning. RAG is context injection. Fine-tuning is model adaptation. For best results, combine both.
Common Questions About Local RAG
How many documents can local RAG handle?
Chroma handles 100K–1M documents on consumer hardware. Qdrant scales to billions with distributed setup. Beyond 1M, use Qdrant or Milvus.
What latency should I expect?
Embedding query (nomic-embed-text on CPU): 50–200ms. Retrieval (Chroma on disk): 10–50ms. LLM generation: 2–10 seconds (depends on model size). Total: 2–10 seconds per query.
Can RAG handle real-time document updates?
Yes. Add new documents to the vector DB dynamically. Indexing latency is 100–500ms per document, so real-time updates are practical.
Is local RAG cheaper than cloud APIs?
Yes. No per-token cost, no API calls to external services. One-time setup of embeddings, then free queries.
Can I use cloud embeddings with local LLMs?
Yes. Use OpenAI, Cohere, or other cloud embeddings for indexing, then use local LLMs for generation. Hybrid approach.
Sources
- LlamaIndex Documentation — docs.llamaindex.ai
- LangChain RAG Guide — python.langchain.com/docs/use_cases/question_answering
- Chroma Documentation — docs.trychroma.com
- Qdrant Vector Search Engine — qdrant.tech
- RAG Paper (Lewis et al.) — arxiv.org/abs/2005.11401