RAG chunk sizes are tuned for retrieval quality vs. token cost

RAG chunk size controls how much text is inserted into the LLM context per retrieved document. Oversized chunks (full pages at 3,000+ tokens each) flood the context window with surrounding noise, reducing the signal-to-noise ratio and leaving less room for conversation history. High topK values compound this: retrieving 20 chunks at 2,000 tokens each consumes 40,000 tokens before the user's question is even accounted for. OWASP LLM10 flags uncontrolled context consumption. Poorly tuned RAG raises both cost and error rate simultaneously.

Tune chunk size to 500–1,500 tokens with 10–20% overlap, and bound topK to 3–10 depending on average chunk relevance. Start conservative and increase only if retrieval quality metrics warrant it.

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,   // ~750 tokens — good balance of context and precision
  chunkOverlap: 150, // preserves context across chunk boundaries
});

const chunks = await splitter.splitDocuments(documents);

// Keep topK bounded in all vector retrieval calls
const results = await vectorStore.similaritySearch(query, 5);

Verify by examining the assembled prompt context for a typical query — the retrieved content should be directly relevant to the question, with no padding from surrounding document sections.