Skip to content

[readme] Update readme #81

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Jintao-Huang merged 4 commits into from
May 15, 2026
Merged

[readme] Update readme #81

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6da74a7
update readme
Jintao-Huang May 15, 2026
a605ff2
fix
Jintao-Huang May 15, 2026
be208c6
fix
Jintao-Huang May 15, 2026
fcb0988
fix
Jintao-Huang May 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 63 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,32 @@ You can contact us and communicate with us by adding our group:
|:-------------------------:|
| <img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/megatron.png" width="200" height="200"> |

## 📝 Introduction

**mcore-bridge** is a large language model and multimodal large model definition library built on the Megatron-Core ecosystem, developed by the ModelScope community. It currently supports 300+ text-only models and 200+ multimodal models, including large language models such as Qwen3-Next, GLM5.1, DeepSeek-V3.2, Minimax2.7, Kimi K2.5, and GPT-OSS, as well as multimodal large models such as Qwen3.5-VL, Qwen3-Omni, GLM4.6-V, InternVL3.5, and Ovis2.5.

------

**Why Choose mcore-bridge?**

- **Model Coverage**: Supports 300+ text-only large language models and 200+ multimodal large models, with Day 0 support for popular models.
- **Hardware Support**: Compatible with a wide range of hardware platforms, including A10/A100/H100/B200, RTX series, and domestic hardware such as Ascend NPU.
- **Training Methods**: Supports both full-parameter training and LoRA training, with compatibility with the PEFT ecosystem.
- **Parallelism Techniques**: Supports multiple parallelism strategies provided by Megatron-Core, including tensor parallelism, pipeline parallelism, sequence parallelism, context parallelism, expert parallelism, and virtual pipeline parallelism.
- **Multimodal Capabilities**: Supports multimodal FP8 training, MTP, sequence padding-free, and packing features.
- **Task Types**: Supports a variety of task types, including Causal LM, sequence classification, Embedding, and Reranker.
- **Ecosystem Compatibility**: Supports direct loading and saving of LoRA/full-parameter safetensors weights, with compatibility with mainstream inference frameworks such as Transformers, vLLM, and SGLang.

------

**Related Documentation:**

- [Integrating Mcore-Bridge with ms-swift](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Mcore-Bridge.html)
- [Supported Models List](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html)
- [Custom Megatron Models](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Custom-Model.html)
- [Qwen3.5 Training Best Practices](https://swift.readthedocs.io/en/latest/BestPractices/Qwen3_5-Best-Practice.html)


## 🎉 News
- 🎉 2026.03.30: MCore-Bridge is released! Providing Megatron-Core model definitions for state-of-the-art large models and making Megatron training as simple as Transformers.

Expand All @@ -75,28 +101,58 @@ pip install -e .
uv pip install -e . --torch-backend=auto
```

Recommended Runtime Environment:

| | Range | Recommended | Notes |
|--------------|--------------|-------------|--------------------|
| python | >=3.10 | 3.12 | |
| cuda | | cuda12.8/13.0 | |
| torch | >=2.0 | 2.8.0/2.11.0 | |
| transformer-engine | >=2.3 | 2.14.1 | |
| apex | | 0.1 | |
| megatron-core | >=0.15,<0.18 | 0.17.0 | |
| flash-attn | | 2.8.3/3.0.0b1 | |
| transformers | >=4.33 | 4.57.6/5.8.1 | |
| modelscope | >=1.23 | | |
| peft | >=0.11,<0.20 | | LoRA |


## ✨ Model List

The following is the list of models supported by MCore-Bridge:

| Series | model_type |

**text-only large models:**

| Series | model_type |
| -------- | ------------------------------------------------------------ |
| Qwen | qwen2, qwen2_moe<br />qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3, qwen3_moe<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr<br />qwen3_next, qwen3_5, qwen3_5_moe |
| Qwen | qwen2, qwen2_moe<br />qwen3, qwen3_moe, qwen3_next |

@gemini-code-assist gemini-code-assist Bot May 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The models qwen3_5 and qwen3_5_moe appear to be misclassified in the multimodal table. Based on the naming convention and their placement in the previous version of the README, they should likely be listed here under text-only models.

Suggested change
| Qwen | qwen2, qwen2_moe<br />qwen3, qwen3_moe, qwen3_next |
| Qwen | qwen2, qwen2_moe<br />qwen3, qwen3_moe, qwen3_next, qwen3_5, qwen3_5_moe |

| DeepSeek | deepseek_v3, deepseek_v32 |
| GLM | glm4, glm4_moe, glm4_moe_lite<br />glm4v, glm4v_moe, <br />glm_moe_dsa |
| GLM | glm4, glm4_moe, glm4_moe_lite<br />glm_moe_dsa |
| MiniMax | minimax_m2 |
| Kimi | kimi_k2, kimi_vl, kimi_k25 |
| Kimi | kimi_k2, kimi_k25 |
| Bailing | bailing_moe |
| InternLM | internlm3, internvl_chat, internvl |
| Ovis | ovis2_5 |
| Llama | llama, llama4 |
| InternLM | internlm3 |
| Llama | llama |
| GPT-OSS | gpt_oss |
| Hunyuan | hy_v3 |
| ERNIE | ernie4_5, ernie4_5_moe |
| MiMo | mimo |
| Dots | dots1 |
| OLMoE | olmoe |

**multimodal large models:**
| Series | model_type |
| -------- | ------------------------------------------------------------ |
| Qwen | qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr<br />qwen3_5, qwen3_5_moe |

@gemini-code-assist gemini-code-assist Bot May 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The model types qwen3_5 and qwen3_5_moe appear to be text-only models and should likely be listed in the 'text-only large models' table instead of the multimodal one, especially since qwen3_next is correctly placed in the text list. If these refer to multimodal versions, consider renaming them to qwen3_5_vl and qwen3_5_moe_vl for consistency with other multimodal entries like qwen2_vl and qwen3_vl.

@gemini-code-assist gemini-code-assist Bot May 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

As noted in the text-only table, qwen3_5 and qwen3_5_moe should be moved to the text-only section. Additionally, the introduction mentions Qwen3.5-VL, which is missing from this list if it differs from qwen3_5.

Suggested change
| Qwen | qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr<br />qwen3_5, qwen3_5_moe |
| Qwen | qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr |

| GLM | glm4v, glm4v_moe |
| Kimi | kimi_vl |
| InternVL | internvl_chat, internvl |
| Ovis | ovis2_5 |
| Llama | llama4 |
| Llava | llava-onevision |
Comment on lines +127 to +153

@gemini-code-assist gemini-code-assist Bot May 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There are several inconsistencies between the models mentioned in the Introduction and those listed in the Model List tables: - Text-only: GLM5.1 and Minimax2.7 are mentioned in the intro but missing from the table. - Multimodal: GLM4.6-V and InternVL3.5 are mentioned in the intro but missing from the table (only glm4v and internvl are listed). Please update the tables to include these models with their respective model_type identifiers.




## 🚀 Quick Start

Expand Down
68 changes: 61 additions & 7 deletions README_zh.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,31 @@
|:-------------------------:|
| <img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/megatron.png" width="200" height="200"> |

## 📝 简介

**mcore-bridge** 是由魔搭社区推出的、基于 Megatron-Core 生态构建的大模型与多模态大模型定义库。目前已支持 300+ 纯文本模型与 200+ 多模态模型。其中大语言模型包括 Qwen3-Next、GLM5.1、DeepSeek-V3.2、Minimax2.7、Kimi K2.5、GPT-OSS 等；多模态大模型包括 Qwen3.5-VL、Qwen3-Omni、GLM4.6-V、InternVL3.5、Ovis2.5 等。

------

**为什么选择 mcore-bridge？**

- **模型类型**：支持 300+ 纯文本大模型与 200+ 多模态大模型，热门模型 Day 0 支持。
- **硬件支持**：支持 A10/A100/H100/B200、RTX 系列、以及国产硬件昇腾 NPU 等多种硬件平台。
- **训练方式**：支持全参数训练与 LoRA 训练，兼容 PEFT 生态。
- **并行技术**：支持 Megatron Core 提供的多种并行策略（张量并行、流水线并行、序列并行、上下文并行、专家并行、虚拟流水线并行）。
- **多模态能力**：支持多模态 FP8 训练、MTP、序列 padding-free 及 packing 等特性。
- **任务类型**：支持因果语言模型（Causal LM）、序列分类、Embedding 及 Reranker 等多种任务类型。
- **生态兼容**：支持直接加载与保存 LoRA/全参数 safetensors 权重，兼容 Transformers、vLLM、SGLang 等主流推理框架。

------

**相关文档：**
- [ms-swift集成Mcore-Bridge](https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Mcore-Bridge.html)
- [支持的模型列表](https://swift.readthedocs.io/zh-cn/latest/Instruction/Supported-models-and-datasets.html)
- [自定义Megatron模型](https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Custom-Model.html)。
- [Qwen3.5训练最佳实践](https://swift.readthedocs.io/zh-cn/latest/BestPractices/Qwen3_5-Best-Practice.html)


## 🎉 新闻
- 🎉 2026.03.30: MCore-Bridge 正式发布！为最先进的大模型提供 Megatron-Core 模型定义，让 Megatron 训练像 Transformers 一样简单。

Expand All @@ -74,28 +99,57 @@ pip install -e .
uv pip install -e . --torch-backend=auto
```


推荐运行环境：
| | 范围 | 推荐 | 备注 |
|--------------|--------------|-------------|--------------------|
| python | >=3.10 | 3.12 | |
| cuda | | cuda12.8/13.0 | |
| torch | >=2.0 | 2.8.0/2.11.0 | |
| transformer-engine | >=2.3 | 2.14.1 | |
| apex | | 0.1 | |
| megatron-core | >=0.15,<0.18 | 0.17.0 | |
| flash-attn | | 2.8.3/3.0.0b1 | |
| transformers | >=4.33 | 4.57.6/5.8.1 | |
| modelscope | >=1.23 | | |
| peft | >=0.11,<0.20 | | LoRA |


## ✨ 模型列表

以下为MCore-Bridge支持的模型列表：

**纯文本模型：**

| 系列 | model_type |
| -------- | ------------------------------------------------------------ |
| Qwen | qwen2, qwen2_moe<br />qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3, qwen3_moe<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr<br />qwen3_next, qwen3_5, qwen3_5_moe |
| Qwen | qwen2, qwen2_moe<br />qwen3, qwen3_moe, qwen3_next |

@gemini-code-assist gemini-code-assist Bot May 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The models qwen3_5 and qwen3_5_moe should be moved here from the multimodal table to maintain consistency with the text-only classification of these base models.

Suggested change
| Qwen | qwen2, qwen2_moe<br />qwen3, qwen3_moe, qwen3_next |
| Qwen | qwen2, qwen2_moe<br />qwen3, qwen3_moe, qwen3_next, qwen3_5, qwen3_5_moe |

| DeepSeek | deepseek_v3, deepseek_v32 |
| GLM | glm4, glm4_moe, glm4_moe_lite<br />glm4v, glm4v_moe, <br />glm_moe_dsa |
| GLM | glm4, glm4_moe, glm4_moe_lite<br />glm_moe_dsa |
| MiniMax | minimax_m2 |
| Kimi | kimi_k2, kimi_vl, kimi_k25 |
| Kimi | kimi_k2, kimi_k25 |
| Bailing | bailing_moe |
| InternLM | internlm3, internvl_chat, internvl |
| Ovis | ovis2_5 |
| Llama | llama, llama4 |
| InternLM | internlm3 |
| Llama | llama |
| GPT-OSS | gpt_oss |
| Hunyuan | hy_v3 |
| ERNIE | ernie4_5, ernie4_5_moe |
| MiMo | mimo |
| Dots | dots1 |
| OLMoE | olmoe |

**多模态模型：**

| 系列 | model_type |
| -------- | ------------------------------------------------------------ |
| Qwen | qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr<br />qwen3_5, qwen3_5_moe |

@gemini-code-assist gemini-code-assist Bot May 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The model types qwen3_5 and qwen3_5_moe appear to be text-only models and should likely be listed in the '纯文本模型' table instead of the multimodal one. If these refer to multimodal versions, consider renaming them to qwen3_5_vl and qwen3_5_moe_vl for consistency with other multimodal entries like qwen2_vl and qwen3_vl.

@gemini-code-assist gemini-code-assist Bot May 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Remove qwen3_5 and qwen3_5_moe from the multimodal list as they are primarily text models. Also, consider adding qwen3_5_vl if that is the intended multimodal model mentioned in the introduction as Qwen3.5-VL.

Suggested change
| Qwen | qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr<br />qwen3_5, qwen3_5_moe |
| Qwen | qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr |

| GLM | glm4v, glm4v_moe |
| Kimi | kimi_vl |
| InternVL | internvl_chat, internvl |
| Ovis | ovis2_5 |
| Llama | llama4 |
| Llava | llava-onevision |
Comment on lines 123 to +150

@gemini-code-assist gemini-code-assist Bot May 15, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Several models mentioned in the Introduction are missing from the Model List tables in this file as well: - Text-only: Missing GLM5.1 and Minimax2.7. - Multimodal: Missing GLM4.6-V and InternVL3.5. Please ensure the Chinese documentation stays in sync with the introduction and the English version.



## 🚀 快速开始

如何使用MCore-Bridge进行训练可以参考[ms-swift项目](https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Mcore-Bridge.html)。这里介绍如何使用代码方式使用Mcore-Bridge。
Expand Down
Loading