Chunking Text for Embeddings: Where the Cut Lands Decides What You Retrieve
Retrieval systems fail quietly. You embed a corpus, wire up a search, ask a question, and get back something that is topically adjacent but does not answer it. The instinct is to blame the embedding model or the similarity metric. Far more often the problem happened before either: the text was cut into pieces that do not each contain a complete idea, and no amount of clever ranking recovers meaning that was destroyed at split time.
This tool splits text into token-bounded chunks with overlap, cutting at the most meaningful boundary available rather than at a fixed offset. It shows you every boundary, because a bad split is only visible by reading where one chunk ends and the next begins. Everything runs in this browser tab.
Recursive Boundary Splitting
The naive approach is to slice every N tokens. It is fast and it is wrong, because a token boundary has no relationship to a semantic one — you will cut mid-sentence, sometimes mid-word, and the resulting vector describes a fragment nobody wrote.
The method here works down a ladder of separators, using the first one that produces pieces small enough to fit. In order: before a Markdown heading, then blank lines, then any line break, then sentence boundaries, then clause boundaries at commas and semicolons, then whitespace between words. Only if none of those exist — a minified bundle, a single enormous word — does it fall back to cutting by token index.
The practical effect is that structure is preserved when the text has any. A document with headings splits at sections. Prose splits at paragraphs, or at sentences when a paragraph is too long. A wall of unbroken text splits at words. Each chunk ends where a person would have ended it, unless the text offered no such opportunity.
What Overlap Is Actually For
Overlap repeats the tail of one chunk at the head of the next. It exists because a fact and the thing it refers to are often on opposite sides of a boundary. "The migration ran overnight. It failed on the third table." Split between those sentences and the second chunk contains a failure with no subject, while the first contains a migration with no outcome. Neither retrieves usefully. With overlap, the second chunk carries the antecedent.
The cost is duplication: overlapping tokens are stored, embedded and searched more than once. Ten to fifteen percent of the chunk size is the usual compromise, which is where the presets sit. The stats panel reports exactly how many tokens the overlap added, so the tradeoff is a number rather than a feeling.
One guard worth knowing about: an overlap equal to or larger than the chunk size would mean each chunk begins with everything the last one ended with, and the process never advances. That is clamped rather than allowed to hang.
Choosing a Size
Chunk size is a precision-versus-context tradeoff, and there is no universally correct answer:
- Small chunks, around 256 tokens, give sharp matches. A vector over two or three sentences is dominated by what those sentences say. Good for question answering over dense reference material. The risk is chunks that match well but lack the surrounding context needed to actually answer.
- Medium chunks, around 512, are the common default and where most embedding models are happiest. Roughly a substantial paragraph.
- Large chunks, 1024 and up, carry more context per hit but dilute the vector. A chunk covering four topics sits in the average of all four and matches none of them strongly.
- Very large chunks, several thousand tokens, stop being a retrieval unit and become a pagination unit — for feeding a long document through a model in sequential passes rather than searching it.
Four Splitting Strategies
Smart split is the recursive ladder described above, and is right for most prose. Paragraphs packs whole paragraphs and never divides one, which suits text where the paragraph is already the unit of thought and you would rather have uneven chunk sizes than a broken one. Markdown headings gives one chunk per section, which is the natural choice for documentation where a heading names exactly what is below it. Lines packs whole lines, for logs, CSV extracts and lists where each row is independent.
In every mode, any unit that still exceeds the limit is reduced by the recursive splitter before packing, so a strategy choice never produces an oversized chunk.
Counting, and What the Number Means
Tokens are counted with the same tokenizer this site uses to size repositories, from the family OpenAI models use. Counts for Claude and Gemini differ somewhat — different vocabularies segment the same text differently — but they track closely enough to size a chunk against a limit. If your budget is tight, leave headroom rather than treating the number as exact.
Note also that an embedding model's stated limit is a hard truncation point, not a target. Exceed it and the tail is silently dropped, which is a failure mode that produces no error and degrades retrieval in a way that is genuinely hard to diagnose.
Output You Can Feed to Something
Three downloads. Plain text with optional --- Chunk n/N --- headers, for reading and
checking. JSON as an array of objects with index, token count and text, for a script that ingests the
lot. JSONL with one object per line, which is what streaming ingestion and most batch embedding endpoints
expect and what you want when the corpus is too large to hold in memory as a single parsed array.
This pairs naturally with the rest of the site: convert a repository, a PDF, a web page or a transcript into text first, then chunk the result here. Nothing is uploaded at either step.