Add HuggingFace pretrained vocab provider and refactor bench data ite… by crutcher · Pull Request #382 · zspacelabs/wordchipper

crutcher · 2026-04-21T03:07:33Z

…ration

Add huggingface feature and HFVocabProvider for loading tokenizers from HuggingFace Hub
Refactor bench data methods to use .values() instead of .iter() for cleaner iteration
Fix clippy warning by allowing explicit counter loop in datagym vocab.

This is broken still:

crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/gpt2' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/gpt2' --encode`
Error: External("No pre-tokenizer")

codecov · 2026-04-21T03:10:11Z

Codecov Report

❌ Patch coverage is 21.93878% with 153 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...rdchipper/src/pretrained/huggingface/hf_factory.rs	0.00%	138 Missing ⚠️
...hipper/src/pretrained/factory/vocab_description.rs	0.00%	9 Missing ⚠️
crates/wordchipper/src/vocab/pair_vocab.rs	0.00%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

crutcher · 2026-04-22T00:49:28Z

crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'cl100k_base' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/wordchipper-cli cat --model cl100k_base --encode`
15339 1917 198
crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/text-embedding-ada-002' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/text-embedding-ada-002' --encode`
15339 1917 198
crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'openai:gpt2' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wordchipper-cli cat --model 'openai:gpt2' --encode`
31373 995 198
crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/gpt2' --encode
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s
     Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/gpt2' --encode`
31373 995 198

further progress; I have a dirty loader; but no tests.

…ration - Add `huggingface` feature and `HFVocabProvider` for loading tokenizers from HuggingFace Hub - Refactor bench data methods to use `.values()` instead of `.iter()` for cleaner iteration - Fix clippy warning by allowing explicit counter loop in datagym vocab. This is broken still: ``` crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/gpt2' --encode Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/gpt2' --encode` Error: External("No pre-tokenizer") ```

- Add support for ByteLevel and Sequence pre-tokenizers in addition to Split - Implement proper byte-to-unicode mapping for byte-level BPE vocabularies - Filter out special tokens when building span maps - Extract and validate split patterns from pre-tokenizer configurations - Add `bytes_char()` function implementing GPT-2 style byte-to-unicode conversion - Use GPT2_PATTERN as default for bare ByteLevel pre-tokenizers - Improve error messages with detailed context about unsupported configurations ```terminaloutput crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'cl100k_base' --encode Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.19s Running `target/debug/wordchipper-cli cat --model cl100k_base --encode` 15339 1917 198 crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/text-embedding-ada-002' --encode Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/text-embedding-ada-002' --encode` 15339 1917 198 crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'openai:gpt2' --encode Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s Running `target/debug/wordchipper-cli cat --model 'openai:gpt2' --encode` 31373 995 198 crutcher@HeatLamp:~/git/wordchipper$ echo 'hello world' | cargo run -p wordchipper-cli -- cat --model 'hf:Xenova/gpt2' --encode Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.16s Running `target/debug/wordchipper-cli cat --model 'hf:Xenova/gpt2' --encode` 31373 995 198 ```

Will now search hf for pretrained models if there is no schema.

- Add `VocabQuery::to_context()` to build cache context from schema, path, and name - Make `VocabDescription::new()` generic over context and description string types - Extract `vocab_from_hf_tokenizer()` as reusable conversion function - Add special token handling to span_config instead of filtering them out - Fix ByteLevel pre-tokenizer to require `use_regex=true` and use OA_GPT2_PATTERN - Update HF vocab description context to use `to_context()` helper - Add tests for `to_context()` with various query formats

This is broken; the vocab seems to have aggregate tokens with no prior piecewise merges that are not special tokens. Which wordchipper enforces is never the case. I need to investigate if this is true for this dataset; and explore what needs to change about wordchipper to handle this. It may be that we need to support a situation where there multi-byte strings in the span map that are not in the byte map, but also *products* in the merge map (so they can only be produced by exact-match lookup, never by merges); but can still be members of merge pairs. ```terminaloutput echo 'hello world' | RUST_BACKTRACE=1 cargo run -p wordchipper-cli -- cat --model Qwen/Qwen3.5-9B --encode Blocking waiting for file lock on artifact directory Compiling wordchipper v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper) Compiling wordchipper-training v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-training) Compiling wordchipper-cli-util v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-cli-util) Compiling wordchipper-cli v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-cli) Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.19s Running `target/debug/wordchipper-cli cat --model Qwen/Qwen3.5-9B --encode` Debug: Some(("ĠÐ°ÑįÑĢÐ¾", 157513)) Debug: { 248067: AddedToken { content: "</tool_response>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248064: AddedToken { content: "<|repo_name|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248046: AddedToken { content: "<|im_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248059: AddedToken { content: "</tool_call>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248054: AddedToken { content: "<|vision_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248044: AddedToken { content: "<|endoftext|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248058: AddedToken { content: "<tool_call>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248049: AddedToken { content: "<|box_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248057: AddedToken { content: "<|video_pad|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248056: AddedToken { content: "<|image_pad|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248062: AddedToken { content: "<|fim_suffix|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248050: AddedToken { content: "<|box_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248055: AddedToken { content: "<|vision_pad|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248048: AddedToken { content: "<|object_ref_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248045: AddedToken { content: "<|im_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248066: AddedToken { content: "<tool_response>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248068: AddedToken { content: "<think>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248069: AddedToken { content: "</think>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248052: AddedToken { content: "<|quad_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248053: AddedToken { content: "<|vision_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248060: AddedToken { content: "<|fim_prefix|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248047: AddedToken { content: "<|object_ref_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248051: AddedToken { content: "<|quad_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248061: AddedToken { content: "<|fim_middle|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248063: AddedToken { content: "<|fim_pad|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248065: AddedToken { content: "<|file_sep|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, } thread 'main' (852780) panicked at crates/wordchipper/src/vocab/span_vocab.rs:247:51: called `Result::unwrap()` on an `Err` value: VocabConflict("Pair (157513, 157853) -> 229601 parent 157513 is not defined") stack backtrace: 0: __rustc::rust_begin_unwind at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/panicking.rs:689:5 1: core::panicking::panic_fmt at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/panicking.rs:80:14 2: core::result::unwrap_failed at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/result.rs:1867:5 3: core::result::Result<T,E>::unwrap at /home/crutcher/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1233:23 4: wordchipper::vocab::span_vocab::SpanMapVocab<T>::to_pair_vocab at ./crates/wordchipper/src/vocab/span_vocab.rs:247:51 5: wordchipper::vocab::unified_vocab::UnifiedTokenVocab<T>::from_span_vocab at ./crates/wordchipper/src/vocab/unified_vocab.rs:85:37 6: wordchipper::pretrained::huggingface::hf_factory::vocab_from_hf_tokenizer at ./crates/wordchipper/src/pretrained/huggingface/hf_factory.rs:199:18 7: <wordchipper::pretrained::huggingface::hf_factory::HFVocabProvider as wordchipper::pretrained::factory::vocab_provider::VocabProvider>::load_vocab at ./crates/wordchipper/src/pretrained/huggingface/hf_factory.rs:245:29 8: wordchipper::pretrained::factory::vocab_factory::VocabFactory::load_vocab at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:229:28 9: wordchipper::pretrained::factory::vocab_factory::load_vocab::{{closure}} at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:99:55 10: wordchipper::pretrained::factory::vocab_factory::with_vocab_factory at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:76:5 11: wordchipper::pretrained::factory::vocab_factory::load_vocab at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:99:5 12: wordchipper_cli_util::model_selector::ModelSelectorArgs::load_vocab at ./crates/wordchipper-cli-util/src/model_selector.rs:29:21 13: wordchipper_cli_util::model_selector::ModelSelectorArgs::load_tokenizer at ./crates/wordchipper-cli-util/src/model_selector.rs:41:26 14: wordchipper_cli::commands::cat_cmd::CatArgs::run at ./crates/wordchipper-cli/src/commands/cat_cmd.rs:51:45 15: wordchipper_cli::Commands::run at ./crates/wordchipper-cli/src/main.rs:44:39 16: wordchipper_cli::main at ./crates/wordchipper-cli/src/main.rs:18:18 17: core::ops::function::FnOnce::call_once at /home/crutcher/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. ```terminaloutput $ echo 'hello world' | RUST_BACKTRACE=1 cargo run -p wordchipper-cli -- cat --model Qwen/Qwen3.5-9B --encode Blocking waiting for file lock on artifact directory Compiling wordchipper v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper) Compiling wordchipper-training v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-training) Compiling wordchipper-cli-util v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-cli-util) Compiling wordchipper-cli v0.9.1 (/home/crutcher/git/wordchipper/crates/wordchipper-cli) Finished `dev` profile [unoptimized + debuginfo] target(s) in 15.19s Running `target/debug/wordchipper-cli cat --model Qwen/Qwen3.5-9B --encode` Debug: Some(("ĠÐ°ÑįÑĢÐ¾", 157513)) Debug: { 248067: AddedToken { content: "</tool_response>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248064: AddedToken { content: "<|repo_name|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248046: AddedToken { content: "<|im_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248059: AddedToken { content: "</tool_call>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248054: AddedToken { content: "<|vision_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248044: AddedToken { content: "<|endoftext|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248058: AddedToken { content: "<tool_call>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248049: AddedToken { content: "<|box_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248057: AddedToken { content: "<|video_pad|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248056: AddedToken { content: "<|image_pad|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248062: AddedToken { content: "<|fim_suffix|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248050: AddedToken { content: "<|box_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248055: AddedToken { content: "<|vision_pad|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248048: AddedToken { content: "<|object_ref_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248045: AddedToken { content: "<|im_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248066: AddedToken { content: "<tool_response>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248068: AddedToken { content: "<think>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248069: AddedToken { content: "</think>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248052: AddedToken { content: "<|quad_end|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248053: AddedToken { content: "<|vision_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248060: AddedToken { content: "<|fim_prefix|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248047: AddedToken { content: "<|object_ref_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248051: AddedToken { content: "<|quad_start|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: true, }, 248061: AddedToken { content: "<|fim_middle|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248063: AddedToken { content: "<|fim_pad|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, 248065: AddedToken { content: "<|file_sep|>", single_word: false, lstrip: false, rstrip: false, normalized: false, special: false, }, } thread 'main' (852780) panicked at crates/wordchipper/src/vocab/span_vocab.rs:247:51: called `Result::unwrap()` on an `Err` value: VocabConflict("Pair (157513, 157853) -> 229601 parent 157513 is not defined") stack backtrace: 0: __rustc::rust_begin_unwind at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/std/src/panicking.rs:689:5 1: core::panicking::panic_fmt at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/panicking.rs:80:14 2: core::result::unwrap_failed at /rustc/59807616e1fa2540724bfbac14d7976d7e4a3860/library/core/src/result.rs:1867:5 3: core::result::Result<T,E>::unwrap at /home/crutcher/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/result.rs:1233:23 4: wordchipper::vocab::span_vocab::SpanMapVocab<T>::to_pair_vocab at ./crates/wordchipper/src/vocab/span_vocab.rs:247:51 5: wordchipper::vocab::unified_vocab::UnifiedTokenVocab<T>::from_span_vocab at ./crates/wordchipper/src/vocab/unified_vocab.rs:85:37 6: wordchipper::pretrained::huggingface::hf_factory::vocab_from_hf_tokenizer at ./crates/wordchipper/src/pretrained/huggingface/hf_factory.rs:199:18 7: <wordchipper::pretrained::huggingface::hf_factory::HFVocabProvider as wordchipper::pretrained::factory::vocab_provider::VocabProvider>::load_vocab at ./crates/wordchipper/src/pretrained/huggingface/hf_factory.rs:245:29 8: wordchipper::pretrained::factory::vocab_factory::VocabFactory::load_vocab at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:229:28 9: wordchipper::pretrained::factory::vocab_factory::load_vocab::{{closure}} at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:99:55 10: wordchipper::pretrained::factory::vocab_factory::with_vocab_factory at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:76:5 11: wordchipper::pretrained::factory::vocab_factory::load_vocab at ./crates/wordchipper/src/pretrained/factory/vocab_factory.rs:99:5 12: wordchipper_cli_util::model_selector::ModelSelectorArgs::load_vocab at ./crates/wordchipper-cli-util/src/model_selector.rs:29:21 13: wordchipper_cli_util::model_selector::ModelSelectorArgs::load_tokenizer at ./crates/wordchipper-cli-util/src/model_selector.rs:41:26 14: wordchipper_cli::commands::cat_cmd::CatArgs::run at ./crates/wordchipper-cli/src/commands/cat_cmd.rs:51:45 15: wordchipper_cli::Commands::run at ./crates/wordchipper-cli/src/main.rs:44:39 16: wordchipper_cli::main at ./crates/wordchipper-cli/src/main.rs:18:18 17: core::ops::function::FnOnce::call_once at /home/crutcher/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. ```

crutcher · 2026-05-11T06:13:31Z

Issue opened to track the orphan tokens bug/feature: #386

Note, this issue is listed in an error message in the PR.

kylegrover · 2026-05-12T03:44:47Z

Trying some related stuff branched off of this on #387 let me know if it's interesting, feel free to pick into PRs or into your branch as you see fit.

crutcher mentioned this pull request Apr 21, 2026

Python: HF tokenizers compat nice-to-haves (from_file, save, add_tokens, training) #377

Open

crutcher added 6 commits May 10, 2026 23:10

Make hf factory a better pass-through factory.

366dd4e

Will now search hf for pretrained models if there is no schema.

Update partial support for HF with basic errors.

460dacf

crutcher force-pushed the crutcher/hf branch from 6991f19 to 460dacf Compare May 11, 2026 06:11

crutcher marked this pull request as ready for review May 11, 2026 06:13

crutcher enabled auto-merge (squash) May 11, 2026 06:15

Fix no-std build.

acf7aef

kylegrover mentioned this pull request May 12, 2026

qwen 3.5 support and generalized conversion script #381

Closed

Navigation Menu

Uh oh!

Add HuggingFace pretrained vocab provider and refactor bench data ite… - #382

Add HuggingFace pretrained vocab provider and refactor bench data ite…#382
crutcher wants to merge 7 commits into
mainfrom
crutcher/hf

crutcher commented Apr 21, 2026

Uh oh!

codecov Bot commented Apr 21, 2026 •

edited

Loading

Uh oh!

crutcher commented Apr 22, 2026

Uh oh!

crutcher commented May 11, 2026

Uh oh!

kylegrover commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

crutcher commented Apr 21, 2026

Uh oh!

codecov Bot commented Apr 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

crutcher commented Apr 22, 2026

Uh oh!

crutcher commented May 11, 2026

Uh oh!

kylegrover commented May 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Apr 21, 2026 •

edited

Loading