Gemma and Phi-4 support with vocab-aware splitting and lstrip/rstrip tokens

[jakehlee]

First, I did not see contributor guidelines or PR templates, so apologies if this is a long shot to merge.

This started as an effort to support GemmaTokenizer. While adding the Replace normalizer, the Replace and ByteFallback decoders, and byte_encode to bpe fixes compatibility, performance gains are not realized because Gemma is an SPM/metaspace pipeline. The normalizer rewrites every space (" " → "▁") before a no-op space split pre-tokenizer, meaning the entire document hits the BPE engine. The cache never hits and the parallelism never triggers.

The solution is vocabulary-aware bigram splitting, where we build a lookup table of bigram pairs that don't appear in the vocabulary, and only split between pairs that cannot merge. This splits the input back into smaller chunks while maintaining token parity. We add a gate to only apply this to models without ByteLevel in the pre-tokenizer sequence to avoid unnecessary overhead.

The result is Gemma4 compatibility and tokenizer performance gains:

% ./validate_model.sh google/gemma-4-31B-it
=== Validating model: google/gemma-4-31B-it ===

--- RyokoAI/ShareGPT52K ---
   Compiling fastokens v0.2.1 (/-/fastokens)
    Finished `release` profile [optimized] target(s) in 7.94s
     Running `/-/fastokens/target/release/examples/simple_bench google/gemma-4-31B-it --dataset RyokoAI/ShareGPT52K`
[00:02:30] [========================================] 45332/45332 (0s)          
═══════════════════════════════════════════
  Benchmark Summary (45332 samples)
═══════════════════════════════════════════
  Total chars:    766126389
  Total tokens:   203044531
───────────────────────────────────────────
  HF total:        138130.18 ms
  fastokens total:  12069.41 ms
  Speedup:             11.44x
───────────────────────────────────────────
  HF avg/sample:       3.047 ms
  ft avg/sample:       0.266 ms
  HF throughput:        5.55 MB/s
  ft throughput:       63.48 MB/s
═══════════════════════════════════════════
PASS: google/gemma-4-31B-it on RyokoAI/ShareGPT52K

--- zai-org/LongBench-v2 ---
    Finished `release` profile [optimized] target(s) in 0.31s
     Running `/-/fastokens/target/release/examples/simple_bench google/gemma-4-31B-it --dataset zai-org/LongBench-v2`
[00:02:28] [========================================] 503/503 (0s)              
═══════════════════════════════════════════
  Benchmark Summary (503 samples)
═══════════════════════════════════════════
  Total chars:    446089036
  Total tokens:   141728831
───────────────────────────────────────────
  HF total:        143278.87 ms
  fastokens total:   4894.06 ms
  Speedup:             29.28x
───────────────────────────────────────────
  HF avg/sample:     284.849 ms
  ft avg/sample:       9.730 ms
  HF throughput:        3.11 MB/s
  ft throughput:       91.15 MB/s
═══════════════════════════════════════════
PASS: google/gemma-4-31B-it on zai-org/LongBench-v2

There are minor optimizations included, including an ASCII fast-path, but this is the bulk of the improvement.

In addition, I noticed Phi-4 tests were failing. This is due to Phi-4's added tokens that have "lstrip": true and "rstrip": true. Adding these flags restores parity.

% ./validate_model.sh microsoft/phi-4      
=== Validating model: microsoft/phi-4 ===

--- RyokoAI/ShareGPT52K ---
    Finished `release` profile [optimized] target(s) in 0.32s
     Running `/-/fastokens/target/release/examples/simple_bench microsoft/phi-4 --dataset RyokoAI/ShareGPT52K`
[00:03:05] [========================================] 45332/45332 (0s)          
═══════════════════════════════════════════
  Benchmark Summary (45332 samples)
═══════════════════════════════════════════
  Total chars:    766126389
  Total tokens:   208458072
───────────────────────────────────────────
  HF total:        170083.00 ms
  fastokens total:  14982.18 ms
  Speedup:             11.35x
───────────────────────────────────────────
  HF avg/sample:       3.752 ms
  ft avg/sample:       0.330 ms
  HF throughput:        4.50 MB/s
  ft throughput:       51.14 MB/s
═══════════════════════════════════════════
PASS: microsoft/phi-4 on RyokoAI/ShareGPT52K

--- zai-org/LongBench-v2 ---
    Finished `release` profile [optimized] target(s) in 0.31s
     Running `/-/fastokens/target/release/examples/simple_bench microsoft/phi-4 --dataset zai-org/LongBench-v2`
[00:02:07] [========================================] 503/503 (0s)              
═══════════════════════════════════════════
  Benchmark Summary (503 samples)
═══════════════════════════════════════════
  Total chars:    446089036
  Total tokens:   123306201
───────────────────────────────────────────
  HF total:        121326.73 ms
  fastokens total:   5463.09 ms
  Speedup:             22.21x
───────────────────────────────────────────
  HF avg/sample:     241.206 ms
  ft avg/sample:      10.861 ms
  HF throughput:        3.68 MB/s
  ft throughput:       81.66 MB/s
═══════════════════════════════════════════
PASS: microsoft/phi-4 on zai-org/LongBench-v2

=== All datasets passed for microsoft/phi-4 ===

Merging this (or a variant) would be valuable, especially for vLLM. Feel free to make/request any changes for the merge.

Thank you!