Short answer
Short answer: use structure-aware, overlapping chunks, then rerank with section and lesson metadata before an answer is generated. For an online course tutor, a chunk should carry enough context to support a citation, but stay small enough that a reranker can compare several candidates. I use the same rule for logistics search: a delivery exception needs its scan event, not an entire shipment history.
The least complex design that works is a two-pass pipeline. Split at semantic boundaries, attach stable source spans, retrieve a wider candidate set, and rerank against the exact question. Chunking is a retrieval decision, not a preprocessing chore.
Why do chunk boundaries change citation quality?
A citation is only useful when a reader can verify the claim without reconstructing missing context. Splitting a lecture in the middle of a definition can retrieve a sentence that looks relevant while dropping the condition that makes it true. The same failure appears in a logistics index when a status line is separated from the location, timestamp, or exception code.
I keep three boundaries visible in the stored record: the source document span, the instructional unit (lesson, module, or transcript turn), and the retrieval text. The source span is immutable. Retrieval text can be normalized, but it must still point back to a precise range. That separation lets the tutor quote a short passage and link to the lesson instead of pretending that a generated paragraph is a primary source.
A practical starting point is 250–400 tokens with 10–15% overlap for prose, then smaller units for tables, formulas, or quiz answers. Those are starting values, not a benchmark. Your mileage may vary when lessons contain long code listings or bilingual captions. Test the boundary policy against real questions before tuning a number.
One short rule: never split a claim from its qualifier.
A boundary policy that survives messy lessons
First parse the document into typed blocks: heading, paragraph, list, code, table, caption, and transcript turn. A heading starts a new candidate, but it also travels as metadata on every child chunk. Lists stay together when their bullets form one procedure; a table stays intact when a row depends on a column heading. For transcript text, preserve speaker and time offsets.
When a block exceeds the target size, split on sentence boundaries and carry a small overlap. Do not duplicate an entire preceding paragraph just to make the index look safe. Duplication inflates candidate counts and can cause the reranker to return five near-identical passages.
Here is a compact representation and splitter. It deliberately returns source offsets so the answer layer can produce citations that survive re-indexing.
from dataclasses import dataclass from typing import Iterable @dataclass(frozen=True) class Block: kind: str text: str start: int end: int heading: str | None = None def chunk_blocks(blocks: Iterable[Block], target: int = 320, overlap: int = 40): for block in blocks: words = block.text.split() if len(words) <= target: yield block continue step = max(1, target - overlap) for left in range(0, len(words), step): part = words[left:left + target] if not part: break # Offsets are approximate here; production code should map tokens # to the parser's exact character spans. yield Block(block.kind, " ".join(part), block.start, block.end, block.heading) The offset comment matters. A tokenizer-only offset is not a citation. Keep the parser span alongside the embedding text, and make the answer service reject a citation that has no source URL, lesson identifier, and character or time range.
Should an online course tutor use chunk overlap or whole lessons?
Overlap helps when a definition crosses a sentence boundary; whole lessons help when the question asks for a multi-step explanation. Neither is a universal default. Retrieve chunks first, then let the reranker see neighboring context as a separate field. That preserves recall without forcing every prompt to carry a 40-page lesson.
For the logistics case, I use event-centered windows: the exception event, its preceding scan, and the next resolution note are separate chunks connected by a shipment key. A question such as “Why was route 18 held?” should rank the exception note above a generic policy paragraph, while still exposing the adjacent scan as evidence. For a tutor, the equivalent key is lesson and concept: “show the derivation” should find the worked example and its stated assumptions.
The reranker should score question-to-chunk relevance plus citation readiness. Citation readiness can be a simple feature set: source exists, heading is present, span is non-empty, and the chunk is not a duplicate of a higher-ranked result. Log each feature. A high relevance score with a missing span is a retrieval failure, not a generation problem.
What should you measure before changing chunk size?
Start with a small labeled set of tutor questions and logistics queries. Mark the passages that contain the answer and the passages that contain the evidence. Measure recall at the candidate stage, citation precision after reranking, and the rate of answers whose cited span actually entails the claim. Keep these metrics separate; one blended score hides the failure mode. I also keep a deliberately awkward test case in the set: a lesson where the definition is at the bottom of one block and its exception opens the next, plus a shipment record where the exception code is separated from the scan location. The trace should show the original blocks, the overlap decision, each rank change, and the final source span. If a change improves recall but makes the citation point at the generic policy paragraph, it failed the product requirement even if the answer sounds fluent.\n\nUse semantic blocks when authors maintain clean headings and tables. Use sentence windows when transcripts are noisy. Add a parent-child link when answers need both a precise quote and a broader lesson explanation. Keep the policy in version control, because changing boundaries changes ranking behavior and citation URLs.
| Signal | What it catches |
|---|---|
| Candidate recall | Boundary split or overly narrow retrieval |
| Citation precision | Plausible but weak supporting passages |
| Span validity | Broken offsets and stale source IDs |
| Duplicate rate | Excessive overlap crowding out variety |
Trace a few misses end to end. Record the original block, generated chunks, rank positions, and final citations. An HTTP 404 for a source span is a data-integrity alert; it should fail the citation check and trigger re-indexing, not silently fall back to an uncited answer.
There is a cost trade-off. More overlap means more vectors, more reranker work, and more duplicate candidates. Less overlap lowers storage pressure but increases the chance that a qualifier lands elsewhere. I usually spend the first review meeting on retention: keep raw source text and offsets, keep embeddings only for the active index, and define how long deleted lessons remain addressable. The catch is that aggressive deletion makes old tutor answers harder to audit.
This approach isn't suitable when your corpus has no stable source identifiers, or when legal review requires immutable page-level snapshots. In those cases, stick with a document store that preserves signed snapshots and make retrieval point at those snapshots. A smaller index with trustworthy provenance beats a larger index that cannot explain itself.
The design decision is therefore easy to state: boundaries should follow meaning, metadata should preserve provenance, and evaluation should test evidence separately from relevance. Tune overlap only after you can show which failure it fixes.
References
Further reading:
