TokenChunker.findStartIndex silently returns wrong index on lookup failure

[chonknick]

Description

In packages/core/src/token.ts, the private findStartIndex method silently falls back to a previous position when it can't find the chunk text in the original string:

private findStartIndex(text: string, chunkText: string, searchFrom: number): number {
    const index = text.indexOf(chunkText, searchFrom);
    return index !== -1 ? index : searchFrom;  // silent fallback
}

When indexOf returns -1, the method returns searchFrom instead of signaling an error. This produces a Chunk with incorrect startIndex/endIndex values that don't correspond to the actual position in the original text, with no indication to the caller that anything went wrong.

When this triggers

This can happen when Tokenizer.decode(Tokenizer.encode(text)) doesn't perfectly round-trip, which currently occurs for:

• Non-BMP Unicode characters (emoji, etc.) due to the charCodeAt/fromCharCode bug (#N)
• Potentially with HuggingFace tokenizers where decode may normalize whitespace or special tokens

Impact

Users relying on text.slice(chunk.startIndex, chunk.endIndex) to extract the original text range will silently get wrong results. Multiple chunks can end up with the same startIndex, making index-based operations unreliable.

Suggested fix

At minimum, consider logging a warning. Ideally, either:

1. Throw an error indicating the round-trip failure, or
2. Use a byte/character offset tracking approach (like FastChunker does) instead of relying on indexOf

Location

• packages/core/src/token.ts lines 120–123