feat(skippy): expose a model-bound tokenizer facade

[i386]

Closes #1213

What this enables

Skippy consumers can use a reusable, in-process tokenizer capability bound to the already-loaded stage-zero model.

• Adds the publishable skippy-tokenizer contract crate.
• Exposes model identity/provenance, special-token policy, native Vec<i32> token IDs, optional token pieces, bounded batches, deterministic request indexes, and typed per-item errors.
• Uses the native tokenizer sizing call to reject oversized output before allocation.
• Caches the capability per runtime and invalidates retained handles on shutdown.
• Keeps the REST /v1/tokenize route as a compatibility adapter, accepting both special_tokens and legacy add_special requests.
• Preserves the old skippy_protocol::tokenizer namespace through a compatibility re-export.
• Registers the new crate in CI, Docker staging, affected-crate planning, Clippy planning, and publish ordering.

Rust API example

In-process consumers use the tokenizer capability already bound to the loaded Skippy runtime:

use skippy_server::SkippyRuntimeHandle;
use skippy_tokenizer::{
    SpecialTokenPolicy, TokenizeRequest, Tokenizer,
};

fn tokenize_prompt(
    runtime: &SkippyRuntimeHandle,
) -> anyhow::Result<Vec<i32>> {
    let tokenizer = runtime.tokenizer_capability()?;
    let request = TokenizeRequest::with_special_tokens(
        tokenizer.identity().clone(),
        "hello".to_owned(),
        SpecialTokenPolicy::Omit,
    );

    let response = tokenizer.tokenize(request)?;
    Ok(response.token_ids)
}

Batch consumers receive deterministic request indexes and per-item results:

let requests = prompts
    .into_iter()
    .map(|text| {
        TokenizeRequest::with_special_tokens(
            tokenizer.identity().clone(),
            text,
            SpecialTokenPolicy::Omit,
        )
    })
    .collect::<Vec<_>>();

let results = tokenizer.tokenize_batch(&requests)?;
for item in results {
    match item.result {
        Ok(response) => println!("{}: {:?}", item.request_index, response.token_ids),
        Err(error) => eprintln!("{}: {error}", item.request_index),
    }
}

The capability uses the already-loaded stage-zero model; it does not perform an HTTP round trip or load a second model. The /v1/tokenize route remains available as a compatibility adapter.

Compatibility

No Skippy ABI or llama.cpp patch changes are included. This does not need to stack on #1194.

Legacy identities without the newly optional tokenizer-version or serving-profile fields continue to match the bound runtime identity.

Validation

• cargo test -p skippy-tokenizer --lib
• cargo test -p skippy-server --lib (387 passed)
• cargo test -p skippy-protocol --lib (45 passed)
• cargo check -p mesh-llm
• cargo clippy -p skippy-tokenizer --all-targets -- -D warnings
• cargo clippy -p skippy-server --all-targets -- -D warnings
• cargo run -p xtask -- repo-consistency ci-crate-lists
• cargo run -p xtask -- repo-consistency publish-crates
• just build

The repository-wide cargo clippy -p mesh-llm --all-targets -- -D warnings remains blocked by 28 pre-existing unfulfilled_lint_expectations warnings in mesh-llm-host-runtime, unrelated to this change.

Summary by CodeRabbit

• New Features

  • Added a shared, model-bound tokenizer for in-process and HTTP use.
  • Added batch tokenization with identity validation, special-token options, token limits, and structured errors.
  • Added optional token pieces and legacy request compatibility.
  • Added bounded tokenization to prevent excessive output allocation.
  • Added lifecycle-aware access that safely handles shutdown and in-flight operations.

• Documentation

  • Documented tokenizer capabilities, limits, compatibility behavior, and runtime requirements.

[github-actions]

This pull request is currently a draft. Reviews will not take place until the PR is marked as ready for review.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8f70d6bd-d44a-4955-8843-df3724a1a46c

📥 Commits

Reviewing files that changed from the base of the PR and between 2717fb8 and 022f98c.

📒 Files selected for processing (2)

• crates/mesh-llm-host-runtime/src/runtime/local_split/tests.rs
• crates/skippy-server/src/tokenizer.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• crates/skippy-server/src/tokenizer.rs

---

📝 Walkthrough

Walkthrough

Adds the skippy-tokenizer contract crate, re-exports it through skippy-protocol, and implements bounded batch tokenization in skippy-server using the loaded stage-zero runtime. Lifecycle checks, structured errors, compatibility HTTP handling, tests, and workspace tooling are updated.

Changes

Tokenizer capability

Layer / File(s)	Summary
Shared tokenizer contract Cargo.toml, crates/skippy-tokenizer/*, crates/skippy-protocol/...	Adds shared identity, request, response, limit, error, batch, and Tokenizer trait types. Preserves legacy add_special compatibility and re-exports the contract through skippy-protocol.
Bounded native tokenization crates/skippy-runtime/src/native.rs	Adds tokenize_bounded and prevents allocation or completion when the native output exceeds the requested token limit.
Server capability and lifecycle crates/skippy-server/...	Implements batch tokenization over the loaded stage-zero runtime. Adds identity validation, input and batch limits, structured errors, special-token handling, HTTP compatibility mapping, lifecycle checks, caching, shutdown handling, and tests.
Workspace and release integration .github/workflows/docker-precheck.yml, scripts/*.sh, crates/skippy-tokenizer/README.md	Registers the new crate in Docker validation, affected-crate reporting, clippy planning, publishing order, and contract documentation.
Runtime test execution support crates/mesh-llm-host-runtime/src/runtime/local_split/tests.rs	Runs the partial-load-failure test in a dedicated thread with an enlarged stack and a current-thread Tokio runtime.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Consumer
  participant TokenizerCapability
  participant StageZeroRuntime
  participant StageModel
  Consumer->>TokenizerCapability: Submit tokenization batch
  TokenizerCapability->>TokenizerCapability: Validate identity and limits
  TokenizerCapability->>StageZeroRuntime: Check runtime activity
  StageZeroRuntime->>StageModel: Request bounded tokenization
  StageModel-->>TokenizerCapability: Return token IDs or bounded miss
  TokenizerCapability-->>Consumer: Return indexed responses or typed errors

Loading

Possibly related PRs

• Mesh-LLM/mesh-llm#1147: Evolves the tokenizer capability into the shared contract used by this PR.
• Mesh-LLM/mesh-llm#1148: Shares tokenizer integration changes in skippy-protocol and skippy-server.
• Mesh-LLM/mesh-llm#1168: Extends related tokenizer behavior in skippy-server.

Suggested reviewers: ndizazzo, michaelneale

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the primary change: exposing a model-bound tokenizer facade.
Linked Issues check	✅ Passed	The changes implement the public facade, model binding, batching, limits, identity checks, lifecycle invalidation, typed errors, and REST compatibility required by issue #1213.
Out of Scope Changes check	✅ Passed	The changes support issue #1213, including crate integration, CI and publishing updates, Docker staging, and related test stabilization.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/issue-1213-tokenizer-facade

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/skippy-server/src/tokenizer.rs`:
- Around line 107-119: Synchronize tokenizer operations with shutdown: in
crates/skippy-server/src/tokenizer.rs lines 107-119, recheck tokenizer_active
while holding the runtime mutex before calling tokenize_bounded; apply the same
guarded recheck in lines 122-140 before detokenize_bytes. In
crates/skippy-server/src/embedded.rs line 286, after clearing tokenizer_active,
acquire and release the runtime mutex before shutdown returns. Add a concurrent
regression test covering a tokenizer request queued behind the runtime mutex
during shutdown.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2f4681c0-8c37-44d4-8435-6dda7a042396

📥 Commits

Reviewing files that changed from the base of the PR and between ab6e9f1 and d0f7b50.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (16)

• .github/workflows/docker-precheck.yml
• Cargo.toml
• crates/skippy-protocol/Cargo.toml
• crates/skippy-protocol/src/lib.rs
• crates/skippy-protocol/src/tokenizer.rs
• crates/skippy-runtime/src/native.rs
• crates/skippy-server/Cargo.toml
• crates/skippy-server/README.md
• crates/skippy-server/src/embedded.rs
• crates/skippy-server/src/tokenizer.rs
• crates/skippy-tokenizer/Cargo.toml
• crates/skippy-tokenizer/README.md
• crates/skippy-tokenizer/src/lib.rs
• scripts/affected-crates.sh
• scripts/plan-clippy-batches.sh
• scripts/publish-crates.sh

💤 Files with no reviewable changes (1)

• crates/skippy-protocol/src/tokenizer.rs

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/skippy-server/src/tokenizer.rs`:
- Around line 558-589: The test
`queued_tokenization_rechecks_lifecycle_after_runtime_lock` must
deterministically pause the worker after its initial lifecycle check and before
locking `runtime`. Add a test-only synchronization signal at that point, have
the test wait for the signal before setting `active` to false, then release the
runtime lock and retain the assertion that tokenization returns
`RuntimeUnavailable`.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f67b9af8-7e83-412a-ac89-be5501892b5c

📥 Commits

Reviewing files that changed from the base of the PR and between d0f7b50 and 2717fb8.

📒 Files selected for processing (2)

• crates/skippy-server/src/embedded.rs
• crates/skippy-server/src/tokenizer.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• crates/skippy-server/src/embedded.rs

[ndizazzo]

Looks clean