Change default drafter architecture from Llama to Qwen3

[orestis-z]

Summary

• Change the default drafter architecture from Llama to Qwen3 in the training script (--draft-arch) and Eagle3 config (transformer_layer_config)
• Qwen3 is now the recommended default for new speculator training

Blocked by

• [Spec Decode] Add Qwen3 architecture support for EAGLE3 vllm#43132

Jira

• INFERENG-8024

Test plan

• Verify --draft-arch defaults to qwen3 when not specified
• Verify Eagle3SpeculatorConfig uses Qwen3Config as default transformer_layer_config
• Run training with default args and confirm Qwen3 layers are used
• Run existing tests

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords (3)

• WIP
• DO NOT MERGE
• DRAFT

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c15f599b-e08f-411e-9e76-7681c032e696

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch qwen3-default-drafter-arch

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[shanjiaz]

Looks good! Let's hold off landing this and wait for Ben's PR on vllm's side to land first?

[rahul-tuli]

LGTM, we'll land once the blocking PR is in

[orestis-z]

Looks good! Let's hold off landing this and wait for Ben's PR on vllm's side to land first?

totally yes!

[benchislett]

On the same note of arch improvement, I recommend adopting the EAGLE3.1 setup: use separate norms on each of the input features before projecting, and use the post-norm hidden state into subsequent draft/TTT steps.

Example checkpoint here: https://huggingface.co/lightseekorg/kimi-k2.6-eagle3.1-mla
Config flags are "fc_norm" and "norm_output".

It is supported in vLLM for both Llama and Qwen3-based EAGLE drafters

[orestis-z]

On the same note of arch improvement, I recommend adopting the EAGLE3.1 setup: use separate norms on each of the input features before projecting, and use the post-norm hidden state into subsequent draft/TTT steps.

Example checkpoint here: https://huggingface.co/lightseekorg/kimi-k2.6-eagle3.1-mla Config flags are "fc_norm" and "norm_output".

It is supported in vLLM for both Llama and Qwen3-based EAGLE drafters

Thanks for your suggestion @benchislett. Totally agree and I have this WIP to support 3.1 setup: #610. And thanks for the checkpoint.

[orestis-z]

@rahul-tuli @shanjiaz no more blockers, ready to land?

[github-actions]

Link Check Results (REPO)

All links are now valid - this issue has been resolved.

---

Marked as resolved: 7b2f765