Chat With 1000+ PDFs Locally: Scaling RAG Beyond Toy Examples

Why Default RAG Breaks Past 1,000 Documents

• Index-out-of-RAM: LanceDB, ChromaDB, and FAISS all start memory-resident. As the index grows past available RAM (typically 5-8 GB of vectors on a 16 GB laptop), they fall back to disk-backed reads and p95 query latency jumps from ~300 ms to 1-3 seconds.
• Cosine-only fails on rare terms: dense embeddings under-weight uncommon proper nouns, drug names, statute numbers, and code identifiers. A query for "Section 230(c)(1)" retrieves chunks about "Section 9" because the embedding cannot distinguish numeric specificity. BM25 catches these; pure cosine misses them.
• Top-K of 4 is too narrow at scale: at 1,000 chunks, top-4 has decent recall. At 50,000 chunks, the truly best chunk is often at rank 12-30 — outside the top-4 window. Retrieval looks like it works (the answers are plausible) but is silently grounded on the wrong passages.
• No metadata filtering wastes the index: asking "what did Smith say about X" on a 10,000-document corpus searches every chunk in the index, when the system should pre-filter to "documents authored by Smith" first. Naive RAG has no concept of metadata pre-filtering.
• Default chunk size of 512/0 fragments long contexts: PDF paragraphs and legal sections rarely fit in 512 tokens. The 0-overlap default loses meaning across boundaries. The 1,000/200 tuning fixes this for medium corpora; hierarchical chunking is needed past 5,000 documents.
• Embedding drift on update: when you add 1,000 new PDFs three months after the original index, sentence-transformer model versions may have changed. Mixing embeddings from two model versions in one index degrades retrieval silently — every architecture forces a full re-index on embedder change.

Architecture Decision Tree: Pick by Corpus Size First

• Under 1,000 documents (under ~5,000 chunks): AnythingLLM Desktop with chunk size 1,000 / overlap 200 and nomic-embed-text-v1.5 as the embedder. No custom code. See the 30-minute step-by-step guide for setup.
• 1,000-5,000 documents (5k-25k chunks): LlamaIndex local mode with hierarchical indices (DocumentSummaryIndex + VectorStoreIndex), Ollama as the LLM provider, nomic-embed-text-v1.5 as the embedder, LanceDB or ChromaDB as the vector store. ~150 lines of Python, runs as a long-lived process.
• 5,000-10,000 documents (25k-50k chunks): Custom stack with Ollama, ChromaDB, BM25 hybrid search via Whoosh or Tantivy, and a BGE-reranker-v2-m3 reranker pass over top-50 candidates. ~300-400 lines of Python. The reranker is non-negotiable at this scale.
• 10,000+ documents (50k+ chunks): Ollama + Qdrant in single-node mode with payload-based metadata filtering, hybrid search using Qdrant native sparse vectors, BGE-reranker-v2-m3, and a hierarchical summary index keyed on document IDs. Production-grade single-user setup.
• Multi-user (any scale): Open WebUI in front of any of the above, OR a small FastAPI wrapper around the same Qdrant + Ollama backend. Multi-user changes the operational story (auth, isolation, rate limiting) but not the retrieval architecture.

Architecture Comparison Table

Option 1: AnythingLLM Tuned (100-1,000 docs)

• LLM: Llama 3.3 8B Q4_K_M via Ollama (5 GB RAM during inference). On 24 GB+ systems, Qwen 2.5 14B Q4 noticeably improves synthesis.
• Embedder: switch from the AnythingLLM Native default to nomic-embed-text-v1.5 via Ollama. The default embedder is the single biggest reason "AnythingLLM does not scale" reports exist.
• Chunking: 1,000 tokens with 200-token overlap, set per workspace under Vector Database settings. The default 512/0 is wrong for any corpus larger than a few dozen documents.
• Top-K: raise from default 4 to 6-8. At 1,000 documents the truly best chunk often sits at rank 5-7, and the LLM can ignore weak chunks better than it can invent missing ones.
• Workspace partitioning: create one workspace per document category (papers, contracts, notes). Each workspace is a separately-indexed LanceDB; cross-workspace queries are not supported, but per-workspace recall is much higher than one giant pool.

Option 2: LlamaIndex Local (1,000-5,000 docs)

• Stack: Ollama + LlamaIndex + LanceDB (or ChromaDB) + nomic-embed-text-v1.5 via the OllamaEmbedding adapter. Persisted to disk; runs as a long-lived Python process you talk to via a CLI or small FastAPI wrapper.
• DocumentSummaryIndex on top of VectorStoreIndex: LlamaIndex builds a per-document summary at index time, then retrieval first picks relevant documents (summary search) and only then searches chunks within those documents. This is the cheapest hierarchical retrieval pattern.
• Query routing: RouterQueryEngine sends fact-recall queries to the chunk index and synthesis queries to the summary index. ~30 lines of code; doubles answer quality on long-document corpora.
• Sentence-window retrieval: an optional second index that retrieves a target sentence plus N surrounding sentences. Useful for legal and academic corpora where the answer is one sentence but its meaning depends on the paragraph around it.
• Persistence: index.storage_context.persist(persist_dir=...) saves everything. Reload time on a 5,000-document index is 10-30 seconds on NVMe SSD.

# Minimal LlamaIndex local RAG with hierarchical indices (~30 lines)
from llama_index.core import VectorStoreIndex, DocumentSummaryIndex, SimpleDirectoryReader
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama
from llama_index.core import Settings

Settings.llm = Ollama(model="llama3.3:8b-instruct-q4_K_M", request_timeout=120)
Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text:latest")
Settings.chunk_size = 1000
Settings.chunk_overlap = 200

docs = SimpleDirectoryReader("./pdfs").load_data()

# Summary index for routing + chunk index for retrieval
summary_index = DocumentSummaryIndex.from_documents(docs)
chunk_index = VectorStoreIndex.from_documents(docs)

summary_index.storage_context.persist("./storage/summary")
chunk_index.storage_context.persist("./storage/chunks")

# At query time, route by question type
response = chunk_index.as_query_engine(similarity_top_k=8).query(
    "What sample size did Smith et al. use?"
)
print(response)

Option 3: Custom Ollama + ChromaDB (5,000-10,000 docs)

• Stack: Ollama + ChromaDB (server mode) + Whoosh or Tantivy for BM25 + BGE-reranker-v2-m3 (~570 MB, runs on CPU at 50-100 candidates/sec). Hosted as a single Python process or split into ingest + query workers.
• Hybrid search at retrieval time: run BM25 and dense vector retrieval in parallel, take top-25 from each, deduplicate, then rerank the merged top-50 with the cross-encoder. Final top-K of 6-8 goes to the LLM.
• ChromaDB metadata fields: populate source_filename, page_number, document_type, author, year on every chunk at index time. Filtering at query time (where={"document_type": "contract"}) cuts retrieval search space by 5-10x with no quality loss.
• Batch indexing: ChromaDB embeds in batches of 32-128 chunks. On an RTX 4070 the BGE-reranker is the bottleneck (50-100 candidates/sec on CPU; 400+/sec on GPU).
• Persistence: ChromaDB writes to a SQLite + Parquet directory. A 50,000-chunk index on disk is ~3-5 GB. Backup is a directory copy.

Option 4: Ollama + Qdrant (10,000+ docs)

• Stack: Ollama + Qdrant (Docker, single-node) + native sparse vectors (BM25-equivalent built into Qdrant 1.10+) + BGE-reranker-v2-m3 + a small Python orchestration layer.
• Native hybrid: Qdrant supports dense + sparse vectors in one collection, with weighted fusion at query time. No separate BM25 process to maintain.
• HNSW tuning: at 50,000+ vectors, raise ef_construct to 200 and m to 32 for the index build, and use ef=128 at query time. Defaults work but trade ~10% recall for build speed.
• Payload schemas for filtering: Qdrant treats payloads as first-class. Index author, document_type, year, and tags as keyword payloads to enable sub-millisecond pre-filtering.
• Hierarchical retrieval: maintain two collections — summaries (one vector per document) and chunks (the usual). Route queries through the summary collection first, then chunk-search within the matched document IDs.
• Persistence: Qdrant writes to a single mounted volume. A 100,000-chunk collection is ~6-12 GB on disk depending on payload size and HNSW settings.

# Qdrant collection with dense + sparse vectors and metadata filtering
from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, SparseVectorParams, SparseIndexParams
)

client = QdrantClient(host="localhost", port=6333)

client.create_collection(
    collection_name="docs",
    vectors_config={
        "dense": VectorParams(size=768, distance=Distance.COSINE),  # nomic-embed-text-v1.5
    },
    sparse_vectors_config={
        "bm25": SparseVectorParams(index=SparseIndexParams(on_disk=False)),
    },
)

# Query: hybrid search + payload filter, no separate BM25 process needed
from qdrant_client.models import Filter, FieldCondition, MatchValue, Prefetch

results = client.query_points(
    collection_name="docs",
    query=dense_vec,
    using="dense",
    prefetch=[
        Prefetch(query=sparse_vec, using="bm25", limit=25),
        Prefetch(query=dense_vec, using="dense", limit=25),
    ],
    query_filter=Filter(
        must=[FieldCondition(key="document_type", match=MatchValue(value="contract"))]
    ),
    limit=50,  # before rerank
)

Hybrid Search: BM25 + Vector Beats Either Alone

• Why dense alone fails: embeddings under-weight rare tokens. Queries like "RFC 9110 section 7.4" or "MNDA-2024-0143" get embedded near generic IETF / contract chunks. BM25 catches the exact identifier; dense search misses it.
• Why BM25 alone fails: lexical matching misses paraphrase. A query "How do we cancel?" against a chunk titled "Termination procedures" matches in dense space but scores 0 in BM25.
• Reciprocal Rank Fusion (RRF) is the standard combiner: for each chunk that appears in either result list, score it as 1/(60+rank_dense) + 1/(60+rank_bm25). Sort descending. The 60 is a smoothing constant; values 30-100 work in practice.
• Practical recipe: retrieve top-25 from each method, combine via RRF, take the top-50, send to a reranker, then top-6-8 to the LLM. This is the standard production pipeline at every scale past 1,000 documents.
• Storage cost: BM25 indices are small (~50-150 MB per 10,000 documents) compared to dense indices (~500 MB-2 GB at the same scale). Adding BM25 to an existing dense store is cheap.

Reranking: The Top-N Refinement Pass

• BGE-reranker-v2-m3 (~570 MB, multilingual, Apache 2.0) is the default choice in May 2026. Runs at 50-100 candidates/sec on a modern CPU; 400+ /sec on GPU. Latency cost for top-50 reranking is ~200-500 ms on CPU, ~80-150 ms on GPU.
• Why cross-encoders win on retrieval: dense embeddings encode query and document independently, so the model never sees them together. A cross-encoder reads `[CLS] query [SEP] candidate [SEP]` jointly and scores the pair directly. Recall@5 typically jumps 15-25 points.
• Where to inject the reranker: after hybrid search, before the LLM. Take top-50 from hybrid, rerank to top-6-8, send those to the LLM as context.
• Alternative — Cohere Rerank API: higher quality but requires a cloud call. For fully-local stacks, BGE-reranker-v2-m3 is the practical default. mxbai-rerank-base-v2 is a strong runner-up.
• Skipping the reranker is fine under 1,000 documents: the quality gain does not justify the latency cost. Past 5,000 documents, skipping it leaves ~15-25% of answers grounded on the wrong chunks.

Metadata Filtering: Pre-Narrow the Search Space

• Universal payload fields to populate at index time: source_filename, page_number, document_type (paper / contract / note / wiki), author, year, language, plus any domain-specific tags (e.g., case_number, project_id, client_id).
• Pre-filter at query time: "What did the 2024 Q3 board minutes say about pricing?" → filter document_type=board_minutes AND year=2024 AND quarter=3 first, then vector search within ~12 documents instead of all 10,000.
• Vector store support: Qdrant payloads, Weaviate properties, ChromaDB metadata, and LanceDB schema columns all support filtering. Performance varies — Qdrant payload filtering on indexed fields is sub-millisecond; ChromaDB metadata filtering on >100k chunks can add 50-150 ms.
• Auto-extracting metadata: for legal corpora, a small LLM pass at index time can extract case numbers, dates, and party names per document. Costs ~30 seconds per document on Llama 3.3 8B; runs once per ingest.
• Combine with hybrid search: payload filter narrows the universe → BM25 + dense retrieval inside the filtered set → rerank. The payload filter is the cheapest 5-10x speedup in any large RAG system.

Hierarchical Retrieval: Summary First, Chunks Second

• Per-document summaries: at index time, prompt the LLM to write a 100-200 token summary of each document. Embed those summaries into a separate summaries collection. Cost is ~30-90 seconds per document on Llama 3.3 8B.
• Two-stage retrieval: (1) embed query, search summaries, take top-5 documents; (2) within those 5 documents, retrieve top-8 chunks via hybrid search; (3) rerank if needed; (4) send to LLM.
• When it helps most: synthesis and multi-document queries ("compare how these papers handle X"). Fact recall ("what value did Smith report?") is fine on the chunk index alone — the summary detour adds latency without quality gain.
• Cost trade-off: doubles index storage (summaries are small but the index itself is duplicated infrastructure). Doubles latency for non-routed queries. The win is in noise reduction at 10,000+ documents.
• LlamaIndex builds this in: DocumentSummaryIndex plus RouterQueryEngine is a 30-line implementation. Custom Python with ChromaDB or Qdrant is ~80-120 lines.

Measured Benchmarks at 100, 1,000, and 10,000 Documents

Storage Sizing and Hardware Requirements

• Raw text per 1,000 PDFs (~12 pages each): ~50-150 MB extracted text. Highly variable based on density.
• Vectors at 1,000 docs: ~300-400 MB on disk including HNSW index overhead. ~120-180 MB if you skip the HNSW index and use brute-force search (acceptable under 5,000 docs).
• Vectors at 10,000 docs: ~3-5 GB on disk. HNSW build takes 10-30 minutes on a modern CPU.
• Vectors at 50,000 docs: ~15-25 GB on disk. Index build time is the bottleneck — plan 2-4 hours of one-time CPU work.
• RAM during query: dense retrieval needs ~30-50% of the index in working memory for low-latency queries. A 5 GB index queries comfortably in 8-16 GB RAM with HNSW; brute force needs the full index resident.
• RAM during indexing: spikes to 2-3x the embedding model size (~600 MB for nomic-embed-text) plus per-batch text. 8 GB free RAM is sufficient for the indexing pass.
• GPU vs CPU: embedding throughput is 4-8x faster on a discrete GPU or Apple Silicon. For 10,000+ document one-shot indexing, the GPU saves 1-3 hours. For query-time embedding (one query at a time), CPU is fine.
• Disk type matters: NVMe SSD is the practical floor at 5,000+ documents. SATA SSD adds 30-100% to query latency on cold queries; spinning disk is unusable past ~2,000 documents.

Incremental Indexing and Deduplication

• Hash-based exact dedup at ingest: SHA-256 the raw file bytes. Skip files whose hash is already in the index. Cheap, catches identical files but misses near-duplicates (different OCR passes of the same scan, format conversions).
• Content-hash dedup: SHA-256 the extracted plain text after stripping whitespace. Catches the same document in different file formats. Adds ~5 ms per file at ingest.
• MinHash for near-duplicates: for legal and academic corpora where multiple drafts of the same document accumulate, compute a MinHash signature (~128 bytes per document) and skip files within a Jaccard similarity threshold of an existing entry.
• Document IDs are forever: never reuse a document ID after deletion. Vector stores often retain orphaned vectors briefly; reusing IDs causes silent confusion. Use UUIDs or hash-based IDs.
• Re-embedding on embedder change: every architecture forces a full re-index when you change the embedding model. Plan an embedder choice you will commit to for at least a year before indexing 10,000 documents.
• Deletes: ChromaDB and Qdrant support point deletion by ID. LanceDB requires a compaction pass to reclaim disk space — schedule weekly if you delete more than ~5% of the corpus per month.

Monitoring RAG Quality at Scale

• Build a small golden set: 30-50 queries you know the right answer to, drawn from real use. Include fact recall (5-10), synthesis (5-10), cross-document (5-10), edge cases (5-10), and known-miss queries (5-10) where the answer is not in the corpus.
• Track three metrics per query: retrieval recall (did the right chunk appear in top-K?), generation faithfulness (does the answer match the chunk?), and refusal rate (does the system correctly say "not in corpus" for known-miss queries?).
• Re-run on every meaningful change: new ingest batches, embedder changes, chunk-size changes, prompt tweaks. Diff results against the previous run; flag any query whose retrieval recall or answer changed.
• Trulens or RAGAS for automated eval frameworks. Both run locally and integrate with LlamaIndex. Manual scoring of 30-50 queries is also fine and often more accurate.
• Latency budgets: track p50 and p95 query latency over time. A 50% jump in p95 typically means the index outgrew RAM — the early signal that you need to graduate to the next architecture tier.

FAQ

Related Reading

• AnythingLLM vs PrivateGPT vs Open WebUI: Best Local RAG in 2026 — base context for choosing the desktop RAG platform before scaling.
• Best Embedding Models for Local RAG in 2026 — embedder choice is the single most consequential decision before indexing 10,000 documents.
• Local RAG for Private Business Data — natural escalation when personal-scale RAG meets enterprise compliance requirements.
• Build Local RAG on Your PDFs in 30 Minutes (Ollama + AnythingLLM) — the entry-level setup before you hit the scaling cliff.
• RAG Explained: How to Ground AI Answers in Real Data (2026) — concept authority on what RAG is and why each retrieval component matters.
• Local LLM Hardware Guide 2026 — hardware sizing reference for 10,000+ document corpora.
• Power Local LLM Hub — full guide library for the cluster.