Skip to content

feat: add build module and tokie CLI #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chonknick wants to merge 12 commits into
base: main
Choose a base branch
from
Open

feat: add build module and tokie CLI #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
4c77e5a
docs: add design spec for build module and tokie-cli
chonknick Apr 22, 2026
40edb98
docs: add implementation plan for build module and tokie-cli
chonknick Apr 22, 2026
1018920
feat: add build feature flag, move tokenizers/tiktoken-rs to optional…
chonknick Apr 22, 2026
478a8e2
feat: add build module with convert function
chonknick Apr 22, 2026
207884c
feat: add verify function with auto-detected reference backend
chonknick Apr 22, 2026
378539a
feat: add upload function for tokiers/ HF org
chonknick Apr 22, 2026
0ac3e64
feat: add tokie-cli crate with convert and verify commands
chonknick Apr 22, 2026
1cf2e82
chore: update Cargo.lock for tokie-cli
chonknick Apr 22, 2026
f72b0c8
feat: use enwik8 for verification instead of hardcoded sentences
chonknick Apr 22, 2026
987a8e8
ci: run tokenizer accuracy tests on PRs, update feature gate to build
chonknick Apr 22, 2026
f67ba6d
fix: speed regression CI fails on Cargo.lock conflict during checkout
chonknick Apr 22, 2026
f783c82
fix: verify full text instead of chunks to avoid false BPE mismatches
chonknick Apr 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions .github/workflows/speed-regression.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,11 @@ jobs:
fi
echo "Benchmarking base commit: $BASE_SHA"
git stash --include-untracked || true
git checkout "$BASE_SHA"
git checkout "$BASE_SHA" -- . || git checkout --force "$BASE_SHA"
cargo run --example speed_regression --release --features hf \
> benchmark_base.json 2>benchmark_base.log || echo '{"results":[]}' > benchmark_base.json
cat benchmark_base.log
git checkout -
git stash pop || true
git checkout --force - || true

- name: Compare results
if: github.event_name == 'push' || github.event_name == 'pull_request'
Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/tokenizer-accuracy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ name: Tokenizer Accuracy
on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch:

env:
Expand Down Expand Up @@ -41,4 +43,4 @@ jobs:
fi

- name: Run tokenizer accuracy tests
run: cargo test -p tokie --test accuracy --features hf -- --ignored
run: cargo test -p tokie --test accuracy --features build -- --ignored
103 changes: 98 additions & 5 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 16 additions & 0 deletions crates/tokie-cli/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[package]
name = "tokie-cli"
version = "0.0.1"
edition = "2024"
description = "CLI for building, verifying, and uploading tokie tokenizers"
license = "MIT OR Apache-2.0"
repository = "https://github.com/chonkie-inc/tokie"

[[bin]]
name = "tokie"
path = "src/main.rs"

[dependencies]
tokie = { path = "../tokie", features = ["hf", "build"] }

@coderabbitai coderabbitai Bot Apr 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, check if the files exist and locate them
find . -name "Cargo.toml" -type f | grep -E "(tokie-cli|tokie)" | head -20

Repository: chonkie-inc/tokie

Length of output: 178

🏁 Script executed:

# Check the structure of the repository
ls -la crates/ 2>/dev/null || echo "crates directory structure check"

Repository: chonkie-inc/tokie

Length of output: 399

🏁 Script executed:

# Read the tokie-cli Cargo.toml file
cat -n crates/tokie-cli/Cargo.toml 2>/dev/null || echo "File not found"

Repository: chonkie-inc/tokie

Length of output: 639

🏁 Script executed:

# Read the tokie Cargo.toml file to get its version
cat -n crates/tokie/Cargo.toml 2>/dev/null || echo "File not found"

Repository: chonkie-inc/tokie

Length of output: 3317

Add a versioned dependency spec for tokie to support cargo install from crates.io.

Path-only dependencies cannot be resolved when installing from crates.io. Add version = "0.0.8" alongside the path attribute to enable both local workspace builds and crates.io installation, which is required for the stated cargo install tokie-cli objective.

Suggested manifest change 
-tokie = { path = "../tokie", features = ["hf", "build"] }
+tokie = { version = "0.0.8", path = "../tokie", features = ["hf", "build"] }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
tokie = { path = "../tokie", features = ["hf", "build"] }
tokie = { version = "0.0.8", path = "../tokie", features = ["hf", "build"] }
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@crates/tokie-cli/Cargo.toml` at line 14, Update the tokie dependency
declaration so it includes a version field in addition to the path (e.g., add
version = "0.0.8") so the crate can be installed from crates.io while still
supporting local workspace builds; modify the existing tokie entry (tokie = {
path = "../tokie", features = ["hf", "build"] }) to include the version
attribute.

clap = { version = "4", features = ["derive"] }
hf-hub = { version = "0.4.3", default-features = false, features = ["ureq"] }
Loading
Loading