August 14, 2026 · 2 min read
Fixed-size, semantic, and structure-aware chunking each trade off differently against retrieval quality — here's how to choose.
Retrieval quality in a RAG system is bounded by chunk quality. A strong embedding model over badly-chunked documents will still retrieve badly.
Every chunk is a unit of retrieval. If a chunk mixes two unrelated ideas, a query about either one retrieves noise alongside the signal. If a chunk is too small, it loses the surrounding context needed to answer a question correctly.
The simplest approach: split text every N tokens, usually with overlap.
def fixed_size_chunks(text: str, size: int = 512, overlap: int = 64):
tokens = tokenize(text)
for i in range(0, len(tokens), size - overlap):
yield detokenize(tokens[i : i + size])
Cheap and predictable, but indifferent to sentence and section boundaries — it will happily cut a chunk in the middle of a sentence.
Splitting along document structure (headings, paragraphs, code blocks) keeps semantically related content together and gives each chunk a natural title from its section heading, which also makes it a better citation.
Group sentences by embedding similarity, so a chunk boundary falls where the topic actually shifts rather than at an arbitrary token count. More expensive to compute at ingestion time, but it tends to produce chunks that map more closely to a single coherent idea — which is what retrieval is actually scoring against.
There is no universally correct chunk size. The right choice depends on:
Chunking strategy is a design decision, not a default. Treat it like any other part of the system: propose it, measure it, and revise it against evidence.