Chunking Strategies for RAG: Size, Overlap and Structure

Chunking decides what the retriever can find. Chunks that are too small lose context; chunks that are too large mix several topics and dilute the embedding.

1. Fixed-size chunks with overlap

The simplest approach: split every N tokens or characters and repeat a little text between neighbours so sentences on a boundary aren't lost.

def chunk_text(text, size=800, overlap=150):
    chunks, start = [], 0
    while start < len(text):
        end = min(start + size, len(text))
        chunks.append(text[start:end])
        if end == len(text):
            break
        start = end - overlap
    return chunks

2. Recursive splitting

Try to split on the largest natural boundary first (paragraphs), then sentences, then words, so chunks end in sensible places.

SEPARATORS = ["\n\n", "\n", ". ", " "]

def recursive_split(text, max_len=800, seps=SEPARATORS):
    if len(text) <= max_len or not seps:
        return [text]
    sep, rest = seps[0], seps[1:]
    parts, current = [], ""
    for piece in text.split(sep):
        candidate = f"{current}{sep}{piece}" if current else piece
        if len(candidate) <= max_len:
            current = candidate
        else:
            if current:
                parts.append(current)
            current = piece
    if current:
        parts.append(current)
    # Pieces that are still too long get split on the next separator
    return [p for part in parts for p in recursive_split(part, max_len, rest)]

3. Structure-aware chunking

• Markdown/HTML: split by headings and keep the heading path ("Billing > Refunds") at the top of each chunk.
• Code: split by function or class, not by line count.
• Tables: keep a table in one chunk, or repeat the header row in each piece.
• FAQs: one question and its answer per chunk.

Picking a size

• Start around 300-800 tokens with 10-20% overlap, then tune with an evaluation set.
• Short, factual lookups favour smaller chunks; "explain how X works" questions favour larger ones.
• Store metadata with each chunk: source URL, title, section, last updated date. You'll need it for filters and citations.

A useful trick is "small to big": search over small chunks for precision, then send the surrounding larger section to the LLM for context.