Skip to content

[bugfix] fix peft_format qwen3_5_moe #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Jintao-Huang merged 1 commit into from
Apr 20, 2026
Merged

[bugfix] fix peft_format qwen3_5_moe #43

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/mcore_bridge/bridge/gpt_bridge.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -723,7 +723,7 @@ def _set_moe_state(

def _get_hf_experts_attr(self, is_mtp: bool = False):
# return hf_grouped, is_gate_up
if not is_mtp and not self.config.fp8_param and self.model_type == 'qwen3_5_moe':
if not is_mtp and not self.config.fp8_param and not self._peft_format and self.model_type == 'qwen3_5_moe':

@gemini-code-assist gemini-code-assist Bot Apr 20, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The addition of not self._peft_format correctly ensures that qwen3_5_moe uses non-grouped expert attributes when processing PEFT weights. This is necessary because LoRA adapters are typically stored per-expert in the HuggingFace state dict, whereas the base model might use a grouped layout.

However, note that other models returning True, True at line 733 (such as qwen3_vl_moe, llama4, and gpt_oss) currently lack this check. If these models are used with PEFT, they will incorrectly report hf_grouped=True, which will trigger a ValueError at line 936 during LoRA weight conversion. You should consider applying a similar check to line 733 to ensure PEFT support for those models as well.

return True, True
if self.model_type in {'glm4v_moe', 'kimi_vl', 'qwen3_omni_moe', 'qwen3_5_moe'} or self.llm_model_type in {
'qwen2_moe', 'qwen3_moe', 'deepseek_v2', 'deepseek_v3', 'kimi_k2', 'dots1', 'ernie4_5_moe', 'glm4_moe',
Expand Down
Loading