Skip to content

fix(qwen35): size the TP sampling scratch from the decodable vocab#761

Open
FeathBow wants to merge 1 commit into
openinfer-project:mainfrom
FeathBow:fix/qwen35-tp-selection-vocab
Open

fix(qwen35): size the TP sampling scratch from the decodable vocab#761
FeathBow wants to merge 1 commit into
openinfer-project:mainfrom
FeathBow:fix/qwen35-tp-selection-vocab

Conversation

@FeathBow

Copy link
Copy Markdown
Collaborator

Description

Fixes #760

#676 bounded token selection to the tokenizer's decodable vocabulary and sized every logits arena from selection_vocab, but it did not touch tp_executor.rs, where the tensor-parallel worker still built its SampleScratch from the checkpoint's padded vocab_size — a line that predates it, from #627. select_batch requires the scratch width to equal the logits width, because a mismatch would otherwise be a silent kernel out-of-bounds rather than a clean error, so on any checkpoint whose tokenizer stops short of the padded width every tensor-parallel prefill and decode failed with select_batch: logits vocab 248077 != scratch vocab 248320. Qwen3.5-4B is such a checkpoint, so TP serving was broken for it.

The four tests that cover this path — the two hf_golden_gate TP2 logits gates, e2e_scheduler TP2, and serving_tp2 — are all #[ignore] because they need two CUDA devices, NCCL, and weights, which is why CI stayed green.

After the change the only remaining uses of vocab_size in this crate are the narrowing comparison itself and the checkpoint tensor-shape validation, both of which want the padded width. Qwen3 and Kimi have no equivalent narrowing and are unaffected.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

Signed-off-by: Feathbow <feathbow@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qwen35: TP serving fails because the sampling scratch is sized from the padded vocab

1 participant