Hi everyone, I've published text_chipper, a Rust crate (with Python bindings via maturin) for splitting large text/Markdown files into token-budgeted chunks for RAG pipelines.

Design highlights:

  • Strict token budgets: Every finalized chunk is re-tokenized in the exact form it is returned (including heading-path prefixes), guaranteeing token_count <= max_length. If a single grapheme cluster exceeds the budget, it returns an error instead of emitting an oversized chunk.
  • Grapheme-safe & offset-preserving: Splits never break Unicode grapheme clusters, and output offsets map cleanly back to the original input.
  • Lightweight Markdown block parser: Tracks heading paths and keeps fenced code blocks and list items intact when they fit.
  • Bounded-memory streaming: On a 514.3 MiB file, from_file_stream() grew memory by ~56 MB (vs ~4.6 GB when loading the whole file into a Python string).

Links:

I'd really appreciate feedback from Rust folks on:

  1. API design for custom token counters (currently Rust-only)
  2. Whether the strict-mode recount tradeoff looks reasonable
  3. Anything un-idiomatic in the crate

1 post - 1 participant

Read full topic