count_tokens splits long inputs at byte offsets without checking UTF-8 character boundaries. Any text past roughly 11KB that is dense in 3-byte codepoints raises a pyo3 PanicException ("byte index is not a char boundary") instead of returning a count.
Hit in production on two real inputs: long CJK prose, and a vision-model caption that rendered a masked password field as a ~4000-char • run.
Versions: tokie 0.0.10 / chonkie-core 0.10.2 / chonkie 1.6.8 (latest at time of writing), reproduced via chonkie's public API:
from chonkie import RecursiveChunker
chunker = RecursiveChunker.from_recipe(
"markdown", lang="en",
tokenizer="Qwen/Qwen3-Embedding-8B",
chunk_size=384,
)
chunker.chunk("数据库连接配置说明文档" * 400) # panics
chunker.chunk("•" * 4000) # panicsBoth raise pyo3_runtime.PanicException from count_tokens. Note the panic is a BaseException, not an Exception, so a bare except Exception in a pipeline won't catch it — without a specific guard it costs the whole document.
Expected: the same counts/boundaries HF tokenizers produces. We verified the HF backend counts the same vocab without the byte-slicing issue and produces identical boundaries on benign text (that's our current workaround: catch the panic and retry once on an HF-tokenizers-backed chunker). Presumably the fix is snapping split offsets to char_indices boundaries before slicing.
Happy to provide more repro detail if useful.
count_tokenssplits long inputs at byte offsets without checking UTF-8 character boundaries. Any text past roughly 11KB that is dense in 3-byte codepoints raises a pyo3PanicException("byte index is not a char boundary") instead of returning a count.Hit in production on two real inputs: long CJK prose, and a vision-model caption that rendered a masked password field as a ~4000-char
•run.Versions: tokie 0.0.10 / chonkie-core 0.10.2 / chonkie 1.6.8 (latest at time of writing), reproduced via chonkie's public API:
Both raise
pyo3_runtime.PanicExceptionfromcount_tokens. Note the panic is aBaseException, not anException, so a bareexcept Exceptionin a pipeline won't catch it — without a specific guard it costs the whole document.Expected: the same counts/boundaries HF
tokenizersproduces. We verified the HF backend counts the same vocab without the byte-slicing issue and produces identical boundaries on benign text (that's our current workaround: catch the panic and retry once on an HF-tokenizers-backed chunker). Presumably the fix is snapping split offsets tochar_indicesboundaries before slicing.Happy to provide more repro detail if useful.