qwen 3.5 support and generalized conversion script - #381
Conversation
…tools Implements a high-performance Qwen3.5 tokenization path, achieving up to ~8x faster single-string encoding than tiktoken via a custom Logos lexer. Core Changes: - Added QWEN35_PATTERN and regex-automata variant to pretrained/openai/patterns.rs. - Implemented Qwen35Token Logos DFA lexer in spanners/span_lexers/logos/qwen35.rs. - Handles Letter/Mark runs, single-char digits, and greedy newline absorption. - Delegates contraction splitting to Gpt2Family post-processing. - Registered Qwen3.5 in regex_automata.rs KNOWN_TRANSFORMS for fallback support. - Relaxed UnifiedTokenVocab subset checks to support pruned vocabularies like Qwen. - Added grounding logic to span_vocab.rs to handle non-decomposable BPE tokens. Python Bindings & Tooling: - Added Tokenizer.from_tiktoken_file() for loading custom BPE vocabularies. - Added SpecialFilter API to control special token recognition during encoding. - Included hf_to_tiktoken.py utility for converting HF tokenizer.json to .tiktoken. - Added comprehensive Qwen3.5 benchmarks and unit tests for SpecialFilter. - Updated documentation and book with custom model loading guides.
This introduces a generic custom vocabulary path (converter and loader) for non-OpenAI models, using Qwen 3.5 as the initial proof-of-concept. Core additions & fixes: - Implement custom `.tiktoken` loader and Python `from_tiktoken_file` API. - Fix Python API bug where `options=None` failed to pass to the Rust binding. - Add feature-gated Qwen 3.5 lexer and reference matchers. - Add synthetic custom-loader tests to Python CI to validate the generic path without requiring giant 100MB+ model assets. - Add Rust regression tests locking in the "pair vocab may be a subset of span vocab" behavior. - Fix compilation error in `qwen35.rs` under the `testing` feature by using a `no_std`-safe string conversion. - Fix Python lockfile resolution for `>=3.9` by removing a stray `black` dev dependency. - Fix existing Rust doctest failures related to the `download` feature. - Update `.gitignore` to exclude generated `.tiktoken`/`.json` assets and Windows `.pyd` build artifacts.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
I want this functionality; but I want it at the rust level.
- this could be a specialized json format loader.
- this could be something that loads through the
tokenizerslibrary, and then converts the vocab in-memory.
I want to avoid python-only features; as this is a rust-first crate.
| /// The Qwen3.5 pretrained vocabulary word pattern. | ||
| /// | ||
| /// Shared by all Qwen3.5 model sizes (0.6B–72B). | ||
| pub const QWEN35_PATTERN: ConstRegexPattern = ConstRegexPattern::Fancy(join_patterns!( |
There was a problem hiding this comment.
This should not be in the openai module.
There was a problem hiding this comment.
This should be pulled into a stand-alone PR
| .pytest_cache | ||
|
|
||
| # Local custom-model conversion artifacts | ||
| bindings/python/qwen3.5.tiktoken |
There was a problem hiding this comment.
this should not be polluting the tree. If you have to generate intermediate artifacts; they should be in "target/"; or managed in tmp or a cache directory directly.
|
Being able to load a HF module via named path |
|
See: #377 |
|
#382 is looking pretty close, I'll just start a new branch from there to pull in any other ideas from this, will PR if it seems useful |
2 participants
I wanted to try this out with Qwen 3.5 in a custom setup and this is what I ended up with. Leaving this as a draft until I have more time to review, let me know if it looks interesting or like it's completely missing the plot of how pretrained should be added