normalizers: support Replace and Prepend (fixes gemma-4-31B tokenizer load)#38
Open
itayh-atero wants to merge 4 commits into
Open
normalizers: support Replace and Prepend (fixes gemma-4-31B tokenizer load)#38itayh-atero wants to merge 4 commits into
itayh-atero wants to merge 4 commits into
Conversation
itayh-atero
force-pushed
the
itayh/prepend-replace-normalizers
branch
2 times, most recently
from
22e4ea2 to
c5e28eb
Compare
…ntered on RedHatAI/gemma-4-31B-it-FP8-Dynamic)
maturin/pyo3 here only supports free-threaded on Python 3.14+, so the -i python3.13t step failed and aborted the x86_64 job before the wheel artifact uploaded. We don't need free-threaded wheels (target is regular CPython 3.12; the cp39-abi3 wheel covers it).
itayh-atero
force-pushed
the
itayh/prepend-replace-normalizers
branch
from
c5e28eb to
76bd7f8
Compare
…ecode chain
gemma's tokenizer.json decoder is a Sequence of Replace("▁"->" "),
ByteFallback, Fuse, Strip(" ", 1, 0). DecoderConfig already parsed these,
but the runtime Decoder only implemented ByteLevel/Sequence and stubbed Fuse
as a no-op, so tokenizer construction failed with 'unsupported decoder type:
Replace' (encountered on RedHatAI/gemma-4-31B-it-FP8-Dynamic under vLLM).
- Implement Replace (reusing normalizers::Replace, the inverse direction),
ByteFallback (run-buffered UTF-8 reassembly; one U+FFFD per byte token on
an invalid run), real Fuse (concat to one token), and Strip.
- Fix Fuse: was identity; gemma's Strip runs after Fuse and expects a single
fused token, so a no-op Fuse would strip a leading space from every piece.
- Decoder Replace.pattern: Value -> ReplacePattern to reuse the normalizer's
pattern engine; re-export normalizers::ReplaceError for the build path.
- Add tests/gemma_replace_decoder.rs: byte-for-byte parity with the HF
tokenizers crate on the verbatim gemma decoder chain (metaspace, multi-byte
fallback, emoji, invalid UTF-8, strip-after-fuse, empty input).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PyTokenizer encode hot path held the GIL for the entire Rust tokenization. Under a high-concurrency serving frontend (e.g. vLLM tokenizing hundreds of requests at once) this serializes all encodes on the GIL and blocks the caller's event loop, inflating time-to-first-token while decode is unaffected. A controlled same-node A/B on gemma-4-31B (VLLM_USE_FASTOKENS on vs off) showed ~+130% TTFT p50 / -14% throughput with the GIL held, entirely in the encode/TTFT portion (ITL/TPOT flat). The read lock was already designed for GIL-released concurrent reads (see the TokenizerState doc comment), so wrap the tokenization in py.allow_threads(): acquire the read lock and build the PyEncoding inside the GIL-released closure, and do only the Py::new allocation under the GIL. encode_batch gets the same treatment so rayon parallelizes without pinning the GIL for the whole batch. (Also: tidy the relocated pad_to_multiple_of math to base.div_ceil(m).) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Replace (literal String + Regex patterns) and Prepend normalizers, with a typed ReplacePattern in the JSON parsing.