Megatron 训练glm5.1 出现rotary_interleaved does not work with multi_latent_attention

[congcynthia]

Checklist / 检查清单

• I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。

Question Description / 问题描述

我的环境：
transformers 5.2.0
transformer_engine 2.15.0
megatron-core 0.17.0
mcore_bridge 1.4.0

log 信息：

[INFO:swift] Setting args.eval_iters: 0
[rank0]: Traceback (most recent call last):
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/cli/_megatron/sft.py", line 7, in
[rank0]: megatron_sft_main()
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/megatron/pipelines/train/sft.py", line 97, in megatron_sft_main
[rank0]: return MegatronSft(args).main()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/pipelines/base.py", line 52, in main
[rank0]: result = self.run()
[rank0]: ^^^^^^^^^^
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/megatron/pipelines/train/sft.py", line 70, in run
[rank0]: trainer = self.prepare_trainer()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/megatron/pipelines/train/sft.py", line 37, in prepare_trainer
[rank0]: return MegatronTrainer(self.args, self.template)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/megatron/trainers/base.py", line 69, in init
[rank0]: self.prepare_model()
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/megatron/trainers/base.py", line 185, in prepare_model
[rank0]: self.unwrapped_models = get_mcore_model(args, self.template.config)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/megatron/model/utils.py", line 81, in get_mcore_model
[rank0]: config = get_mcore_model_config(args, hf_config)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/swift/megatron/model/utils.py", line 72, in get_mcore_model_config
[rank0]: config = ModelConfig(**kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "", line 288, in init
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/mcore_bridge/config/model_config.py", line 317, in post_init
[rank0]: super().post_init()
[rank0]: File "/usr/local/python3.12/lib/python3.12/site-packages/megatron/core/transformer/transformer_config.py", line 1732, in post_init
[rank0]: raise ValueError("rotary_interleaved does not work with multi_latent_attention.")
[rank0]: ValueError: rotary_interleaved does not work with multi_latent_attention.

[Jintao-Huang]

试试mcore-bridge main分支/0.4.1

[congcynthia]

试试mcore-bridge main分支/0.4.1

mcore-bridge没有看到0.4.1版本

[Jintao-Huang]

好像还有点问题，稍等下 我修复一下

[Jintao-Huang]

modelscope/mcore-bridge#99