diff --git a/README.md b/README.md index 175afdc..6fc13a4 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,32 @@ You can contact us and communicate with us by adding our group: |:-------------------------:| | | +## 📝 Introduction + +**mcore-bridge** is a large language model and multimodal large model definition library built on the Megatron-Core ecosystem, developed by the ModelScope community. It currently supports 300+ text-only models and 200+ multimodal models, including large language models such as Qwen3-Next, GLM5.1, DeepSeek-V3.2, Minimax2.7, Kimi K2.5, and GPT-OSS, as well as multimodal large models such as Qwen3.5-VL, Qwen3-Omni, GLM4.6-V, InternVL3.5, and Ovis2.5. + +------ + +**Why Choose mcore-bridge?** + +- **Model Coverage**: Supports 300+ text-only large language models and 200+ multimodal large models, with Day 0 support for popular models. +- **Hardware Support**: Compatible with a wide range of hardware platforms, including A10/A100/H100/B200, RTX series, and domestic hardware such as Ascend NPU. +- **Training Methods**: Supports both full-parameter training and LoRA training, with compatibility with the PEFT ecosystem. +- **Parallelism Techniques**: Supports multiple parallelism strategies provided by Megatron-Core, including tensor parallelism, pipeline parallelism, sequence parallelism, context parallelism, expert parallelism, and virtual pipeline parallelism. +- **Multimodal Capabilities**: Supports multimodal FP8 training, MTP, sequence padding-free, and packing features. +- **Task Types**: Supports a variety of task types, including Causal LM, sequence classification, Embedding, and Reranker. +- **Ecosystem Compatibility**: Supports direct loading and saving of LoRA/full-parameter safetensors weights, with compatibility with mainstream inference frameworks such as Transformers, vLLM, and SGLang. + +------ + +**Related Documentation:** + +- [Integrating Mcore-Bridge with ms-swift](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Mcore-Bridge.html) +- [Supported Models List](https://swift.readthedocs.io/en/latest/Instruction/Supported-models-and-datasets.html) +- [Custom Megatron Models](https://swift.readthedocs.io/en/latest/Megatron-SWIFT/Custom-Model.html) +- [Qwen3.5 Training Best Practices](https://swift.readthedocs.io/en/latest/BestPractices/Qwen3_5-Best-Practice.html) + + ## 🎉 News - 🎉 2026.03.30: MCore-Bridge is released! Providing Megatron-Core model definitions for state-of-the-art large models and making Megatron training as simple as Transformers. @@ -75,21 +101,39 @@ pip install -e . uv pip install -e . --torch-backend=auto ``` +Recommended Runtime Environment: + +| | Range | Recommended | Notes | +|--------------|--------------|-------------|--------------------| +| python | >=3.10 | 3.12 | | +| cuda | | cuda12.8/13.0 | | +| torch | >=2.0 | 2.8.0/2.11.0 | | +| transformer-engine | >=2.3 | 2.14.1 | | +| apex | | 0.1 | | +| megatron-core | >=0.15,<0.18 | 0.17.0 | | +| flash-attn | | 2.8.3/3.0.0b1 | | +| transformers | >=4.33 | 4.57.6/5.8.1 | | +| modelscope | >=1.23 | | | +| peft | >=0.11,<0.20 | | LoRA | + + ## ✨ Model List The following is the list of models supported by MCore-Bridge: -| Series | model_type | + +**text-only large models:** + +| Series | model_type | | -------- | ------------------------------------------------------------ | -| Qwen | qwen2, qwen2_moe
qwen2_vl, qwen2_5_vl, qwen2_5_omni
qwen3, qwen3_moe
qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr
qwen3_next, qwen3_5, qwen3_5_moe | +| Qwen | qwen2, qwen2_moe
qwen3, qwen3_moe, qwen3_next | | DeepSeek | deepseek_v3, deepseek_v32 | -| GLM | glm4, glm4_moe, glm4_moe_lite
glm4v, glm4v_moe,
glm_moe_dsa | +| GLM | glm4, glm4_moe, glm4_moe_lite
glm_moe_dsa | | MiniMax | minimax_m2 | -| Kimi | kimi_k2, kimi_vl, kimi_k25 | +| Kimi | kimi_k2, kimi_k25 | | Bailing | bailing_moe | -| InternLM | internlm3, internvl_chat, internvl | -| Ovis | ovis2_5 | -| Llama | llama, llama4 | +| InternLM | internlm3 | +| Llama | llama | | GPT-OSS | gpt_oss | | Hunyuan | hy_v3 | | ERNIE | ernie4_5, ernie4_5_moe | @@ -97,6 +141,18 @@ The following is the list of models supported by MCore-Bridge: | Dots | dots1 | | OLMoE | olmoe | +**multimodal large models:** +| Series | model_type | +| -------- | ------------------------------------------------------------ | +| Qwen | qwen2_vl, qwen2_5_vl, qwen2_5_omni
qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr
qwen3_5, qwen3_5_moe | +| GLM | glm4v, glm4v_moe | +| Kimi | kimi_vl | +| InternVL | internvl_chat, internvl | +| Ovis | ovis2_5 | +| Llama | llama4 | +| Llava | llava-onevision | + + ## 🚀 Quick Start diff --git a/README_zh.md b/README_zh.md index d5e9d55..25069ca 100644 --- a/README_zh.md +++ b/README_zh.md @@ -49,6 +49,31 @@ |:-------------------------:| | | +## 📝 简介 + +**mcore-bridge** 是由魔搭社区推出的、基于 Megatron-Core 生态构建的大模型与多模态大模型定义库。目前已支持 300+ 纯文本模型与 200+ 多模态模型。其中大语言模型包括 Qwen3-Next、GLM5.1、DeepSeek-V3.2、Minimax2.7、Kimi K2.5、GPT-OSS 等;多模态大模型包括 Qwen3.5-VL、Qwen3-Omni、GLM4.6-V、InternVL3.5、Ovis2.5 等。 + +------ + +**为什么选择 mcore-bridge?** + +- **模型类型**:支持 300+ 纯文本大模型与 200+ 多模态大模型,热门模型 Day 0 支持。 +- **硬件支持**:支持 A10/A100/H100/B200、RTX 系列、以及国产硬件昇腾 NPU 等多种硬件平台。 +- **训练方式**:支持全参数训练与 LoRA 训练,兼容 PEFT 生态。 +- **并行技术**:支持 Megatron Core 提供的多种并行策略(张量并行、流水线并行、序列并行、上下文并行、专家并行、虚拟流水线并行)。 +- **多模态能力**:支持多模态 FP8 训练、MTP、序列 padding-free 及 packing 等特性。 +- **任务类型**:支持因果语言模型(Causal LM)、序列分类、Embedding 及 Reranker 等多种任务类型。 +- **生态兼容**:支持直接加载与保存 LoRA/全参数 safetensors 权重,兼容 Transformers、vLLM、SGLang 等主流推理框架。 + +------ + +**相关文档:** +- [ms-swift集成Mcore-Bridge](https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Mcore-Bridge.html) +- [支持的模型列表](https://swift.readthedocs.io/zh-cn/latest/Instruction/Supported-models-and-datasets.html) +- [自定义Megatron模型](https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Custom-Model.html)。 +- [Qwen3.5训练最佳实践](https://swift.readthedocs.io/zh-cn/latest/BestPractices/Qwen3_5-Best-Practice.html) + + ## 🎉 新闻 - 🎉 2026.03.30: MCore-Bridge 正式发布!为最先进的大模型提供 Megatron-Core 模型定义,让 Megatron 训练像 Transformers 一样简单。 @@ -74,21 +99,37 @@ pip install -e . uv pip install -e . --torch-backend=auto ``` + +推荐运行环境: +| | 范围 | 推荐 | 备注 | +|--------------|--------------|-------------|--------------------| +| python | >=3.10 | 3.12 | | +| cuda | | cuda12.8/13.0 | | +| torch | >=2.0 | 2.8.0/2.11.0 | | +| transformer-engine | >=2.3 | 2.14.1 | | +| apex | | 0.1 | | +| megatron-core | >=0.15,<0.18 | 0.17.0 | | +| flash-attn | | 2.8.3/3.0.0b1 | | +| transformers | >=4.33 | 4.57.6/5.8.1 | | +| modelscope | >=1.23 | | | +| peft | >=0.11,<0.20 | | LoRA | + + ## ✨ 模型列表 -以下为MCore-Bridge支持的模型列表: + +**纯文本模型:** | 系列 | model_type | | -------- | ------------------------------------------------------------ | -| Qwen | qwen2, qwen2_moe
qwen2_vl, qwen2_5_vl, qwen2_5_omni
qwen3, qwen3_moe
qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr
qwen3_next, qwen3_5, qwen3_5_moe | +| Qwen | qwen2, qwen2_moe
qwen3, qwen3_moe, qwen3_next | | DeepSeek | deepseek_v3, deepseek_v32 | -| GLM | glm4, glm4_moe, glm4_moe_lite
glm4v, glm4v_moe,
glm_moe_dsa | +| GLM | glm4, glm4_moe, glm4_moe_lite
glm_moe_dsa | | MiniMax | minimax_m2 | -| Kimi | kimi_k2, kimi_vl, kimi_k25 | +| Kimi | kimi_k2, kimi_k25 | | Bailing | bailing_moe | -| InternLM | internlm3, internvl_chat, internvl | -| Ovis | ovis2_5 | -| Llama | llama, llama4 | +| InternLM | internlm3 | +| Llama | llama | | GPT-OSS | gpt_oss | | Hunyuan | hy_v3 | | ERNIE | ernie4_5, ernie4_5_moe | @@ -96,6 +137,19 @@ uv pip install -e . --torch-backend=auto | Dots | dots1 | | OLMoE | olmoe | +**多模态模型:** + +| 系列 | model_type | +| -------- | ------------------------------------------------------------ | +| Qwen | qwen2_vl, qwen2_5_vl, qwen2_5_omni
qwen3_vl, qwen3_vl_moe, qwen3_omni_moe, qwen3_asr
qwen3_5, qwen3_5_moe | +| GLM | glm4v, glm4v_moe | +| Kimi | kimi_vl | +| InternVL | internvl_chat, internvl | +| Ovis | ovis2_5 | +| Llama | llama4 | +| Llava | llava-onevision | + + ## 🚀 快速开始 如何使用MCore-Bridge进行训练可以参考[ms-swift项目](https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Mcore-Bridge.html)。这里介绍如何使用代码方式使用Mcore-Bridge。