[core] first commit

.github/workflows/lint.yaml

@@ -0,0 +1,22 @@
+name: Lint test
+
+on: [push, pull_request]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.12'
+      - name: Install pre-commit hook
+        run: |
+          pip install pre-commit
+      - name: Linting
+        run: pre-commit run --all-files

.github/workflows/publish.yaml

@@ -0,0 +1,29 @@
+name: release
+
+on:
+  push:
+    tags:
+      - 'v**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}-publish
+  cancel-in-progress: true
+
+jobs:
+  build-n-publish:
+    runs-on: ubuntu-22.04
+    #if: startsWith(github.event.ref, 'refs/tags')
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python 3.12
+        uses: actions/setup-python@v2
+        with:
+          python-version: '3.12'
+      - name: Install wheel
+        run: pip install wheel packaging setuptools==69.5.1
+      - name: Build Mcore Bridge
+        run: python setup.py sdist bdist_wheel
+      - name: Publish package to PyPI
+        run: |
+          pip install twine
+          twine upload dist/* --skip-existing -u __token__ -p ${{ secrets.PYPI_API_TOKEN }}

README.md

@@ -39,22 +39,22 @@
 - [News](#-news)
 - [Installation](#%EF%B8%8F-installation)
 - [Quick Start](#-quick-Start)
-- [Usage](#-Usage)
+- [Model List](#-Model-List)
 - [License](#-License)
 
 
 ## ☎ Groups
 
 You can contact us and communicate with us by adding our group:
 
-WeChat Group |
-:-------------------------:
-<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/megatron.png" width="200" height="200">
+| WeChat Group |
+|:-------------------------:|
+| <img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/megatron.png" width="200" height="200"> |
 
 ## 📝 Introduction
 
 ## 🎉 News
-- 🎉 2025.03.23: MCore-Bridge is released! Making Megatron training as simple as Transformers, providing Megatron-Core model definitions for state-of-the-art large language models.
+- 🎉 2025.04.01: MCore-Bridge is released! Providing Megatron-Core model definitions for state-of-the-art large language models and making Megatron training as simple as Transformers.
 
 ## 🛠️ Installation
 To install using pip:
@@ -85,6 +85,7 @@ You need to create the following file (test.py), then run `CUDA_VISIBLE_DEVICES=
 The saved model can be used for inference by referring to the [example code in the model card](https://modelscope.cn/models/Qwen/Qwen3.5-35B-A3B).
 
 ```python
+# test env: transformers==5.2.0 megatron-core==0.16.1
 import os
 import torch
 import torch.distributed as dist
@@ -93,6 +94,7 @@ from modelscope import snapshot_download
 from transformers import AutoConfig, AutoProcessor
 from mcore_bridge import ModelConfig, get_mcore_model, hf_to_mcore_config
 
+is_rank0 = int(os.getenv('RANK')) == 0
 torch.cuda.set_device(f"cuda:{os.getenv('LOCAL_RANK')}")
 dist.init_process_group(backend='nccl')
 TP, PP, EP, ETP = 2, 2, 2, 1
@@ -115,8 +117,6 @@ config = ModelConfig(
     expert_tensor_parallel_size=ETP,
     sequence_parallel=True,
     mtp_num_layers=1,
-    processor=processor,
-    hf_config=hf_config,
     **config_kwargs)
 
 # Create model
@@ -133,12 +133,110 @@ for name, parameter in bridge.export_weights(mg_models):
 # Save weights
 output_dir = 'Qwen3.5-35B-A3B-HF'
 bridge.save_weights(mg_models, output_dir)
-processor.save_pretrained(output_dir)
-hf_config.save_pretrained(output_dir)
+if is_rank0:
+    processor.save_pretrained(output_dir)
+    hf_config.save_pretrained(output_dir)
 ```
 
-## ✨ Usage
+### Using Peft
 
+Mcore-Bridge is fully compatible with [Peft](https://github.com/huggingface/peft) for LoRA training. The following introduces how to use Peft to prepare a PeftModel and save the incremental weights.
+
+You need to create the following file (test.py), then run `CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 test.py`.
+
+```python
+import copy
+import os
+import torch
+import torch.distributed as dist
+from megatron.core import mpu
+from modelscope import snapshot_download
+from peft import LoraConfig, get_peft_model
+from transformers import AutoConfig, AutoProcessor
+
+from mcore_bridge import ModelConfig, get_mcore_model, hf_to_mcore_config, set_random_seed
+
+is_rank0 = int(os.getenv('RANK')) == 0
+torch.cuda.set_device(f"cuda:{os.getenv('LOCAL_RANK')}")
+dist.init_process_group(backend='nccl')
+TP, PP = 2, 2
+mpu.initialize_model_parallel(
+    tensor_model_parallel_size=TP,
+    pipeline_model_parallel_size=PP,
+)
+# To correctly initialize the model randomly (full parameters/LoRA)
+# you need to set the random seed.
+set_random_seed(42)
+
+model_dir = snapshot_download('Qwen/Qwen3.5-4B')
+hf_config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
+config_kwargs = hf_to_mcore_config(hf_config)
+config = ModelConfig(
+    params_dtype=torch.bfloat16,
+    tensor_model_parallel_size=TP,
+    pipeline_model_parallel_size=PP,
+    sequence_parallel=True,
+    **config_kwargs)
+
+# Create model and load weights
+mg_models = get_mcore_model(config)
+bridge = config.bridge
+bridge.load_weights(mg_models, model_dir)
+
+# Prepare PeftModel and load LoRA weights
+# For multimodal models, it is recommended to use regex to specify target_modules
+target_modules = r'^language_model.*\.(in_proj|out_proj|linear_fc1|linear_fc2|linear_qkv|linear_proj)$'
+# When saving as safetensors, you need to store the corresponding HF target_modules
+hf_target_modules = r'^model.language_model.*\.(in_proj_qkv|in_proj_z|in_proj_b|in_proj_a|out_proj|gate_proj|up_proj|down_proj|q_proj|k_proj|v_proj|o_proj)$'
+lora_config = LoraConfig(task_type='CAUSAL_LM', r=8, lora_alpha=32, lora_dropout=0.05, target_modules=target_modules)
+peft_models = [get_peft_model(model, lora_config) for model in mg_models]
+# Optional
+# bridge.load_weights(peft_models, model_dir, peft_format=True)
+
+# Export LoRA weights
+for name, parameter in bridge.export_weights(mg_models, peft_format=True):
+    pass
+
+# Save LoRA weights
+output_dir = 'Qwen3.5-4B-LoRA'
+bridge.save_weights(mg_models, output_dir, peft_format=True)
+if is_rank0:
+    hf_lora_config = copy.copy(lora_config)
+    hf_lora_config.target_modules = hf_target_modules
+    hf_lora_config.save_pretrained(output_dir)
+```
+
+Using the saved LoRA weights:
+
+```python
+from transformers import Qwen3_5ForConditionalGeneration
+from modelscope import snapshot_download
+from peft import PeftModel
+
+model_dir = snapshot_download('Qwen/Qwen3.5-4B')
+model = Qwen3_5ForConditionalGeneration.from_pretrained(model_dir)
+peft_model = PeftModel.from_pretrained(model, 'Qwen3.5-4B-LoRA')
+```
+
+## ✨ Model List
+
+The following is the list of models supported by MCore-Bridge:
+
+| Series     | model_type                                                   |
+| -------- | ------------------------------------------------------------ |
+| Qwen     | qwen2, qwen2_moe<br />qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3, qwen3_moe<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe<br />qwen3_next, qwen3_5, qwen3_5_moe |
+| DeepSeek | deepseek_v3, deepseek_v32                                    |
+| GLM      | glm4, glm4_moe, glm4_moe_lite<br />glm4v, glm4v_moe, <br />glm_moe_dsa |
+| MiniMax  | minimax_m2                                                   |
+| Kimi     | kimi_k2, kimi_vl                                             |
+| InternLM | internlm3, internvl_chat, internvl                           |
+| Ovis     | ovis2_5                                                      |
+| Llama    | llama, llama4                                                |
+| GPT-OSS  | gpt_oss                                                      |
+| ERNIE    | ernie4_5, ernie4_5_moe                                       |
+| MiMo     | mimo                                                         |
+| Dots     | dots1                                                        |
+| OLMoE    | olmoe                                                        |
 
 ## 🏛 License
 

README_zh.md

@@ -39,21 +39,21 @@
 - [新闻](#-新闻)
 - [安装](#%EF%B8%8F-安装)
 - [快速开始](#-快速开始)
-- [如何使用](#-如何使用)
+- [模型列表](#-模型列表)
 - [License](#-license)
 
 ## ☎ 用户群
 
 请扫描下面的二维码来加入我们的交流群：
 
-微信群 |
-:-------------------------:
-<img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/megatron.png" width="200" height="200">
+| 微信群 |
+|:-------------------------:|
+| <img src="https://raw.githubusercontent.com/modelscope/ms-swift/main/docs/resources/wechat/megatron.png" width="200" height="200"> |
 
 ## 📝 简介
 
 ## 🎉 新闻
-- 🎁 2026.03.23: Mcore-Bridge发布，让Megatron训练像transformers一样简单，为最先进的大语言模型提供 Megatron-Core 模型定义。
+- 🎉 2026.04.01: MCore-Bridge 正式发布！为最先进的大语言模型提供 Megatron-Core 模型定义，让 Megatron 训练像 Transformers 一样简单。
 
 ## 🛠️ 安装
 使用pip进行安装：
@@ -84,6 +84,7 @@ uv pip install -e . --torch-backend=auto
 保存的模型，可以参考[模型卡片的示例代码](https://modelscope.cn/models/Qwen/Qwen3.5-35B-A3B)进行推理。
 
 ```python
+# test env: transformers==5.2.0 megatron-core==0.16.1
 import os
 import torch
 import torch.distributed as dist
@@ -92,6 +93,7 @@ from modelscope import snapshot_download
 from transformers import AutoConfig, AutoProcessor
 from mcore_bridge import ModelConfig, get_mcore_model, hf_to_mcore_config
 
+is_rank0 = int(os.getenv('RANK')) == 0
 torch.cuda.set_device(f"cuda:{os.getenv('LOCAL_RANK')}")
 dist.init_process_group(backend='nccl')
 TP, PP, EP, ETP = 2, 2, 2, 1
@@ -114,8 +116,6 @@ config = ModelConfig(
     expert_tensor_parallel_size=ETP,
     sequence_parallel=True,
     mtp_num_layers=1,
-    processor=processor,
-    hf_config=hf_config,
     **config_kwargs)
 
 # 创建模型
@@ -132,11 +132,108 @@ for name, parameter in bridge.export_weights(mg_models):
 # 保存权重
 output_dir = 'Qwen3.5-35B-A3B-HF'
 bridge.save_weights(mg_models, output_dir)
-processor.save_pretrained(output_dir)
-hf_config.save_pretrained(output_dir)
+if is_rank0:
+    processor.save_pretrained(output_dir)
+    hf_config.save_pretrained(output_dir)
 ```
 
-## ✨ 如何使用
+### 使用Peft
+
+Mcore-Bridge完全兼容使用[Peft](https://github.com/huggingface/peft)进行LoRA训练。以下介绍如何使用peft准备PeftModel，并保存增量权重。
+
+你需要创建以下文件（test.py），然后运行`CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 test.py`。
+
+```python
+import copy
+import os
+import torch
+import torch.distributed as dist
+from megatron.core import mpu
+from modelscope import snapshot_download
+from peft import LoraConfig, get_peft_model
+from transformers import AutoConfig, AutoProcessor
+
+from mcore_bridge import ModelConfig, get_mcore_model, hf_to_mcore_config, set_random_seed
+
+is_rank0 = int(os.getenv('RANK')) == 0
+torch.cuda.set_device(f"cuda:{os.getenv('LOCAL_RANK')}")
+dist.init_process_group(backend='nccl')
+TP, PP = 2, 2
+mpu.initialize_model_parallel(
+    tensor_model_parallel_size=TP,
+    pipeline_model_parallel_size=PP,
+)
+# 为了正确随机初始化模型（全参数/LoRA），你需要设置随机种子
+set_random_seed(42)
+
+model_dir = snapshot_download('Qwen/Qwen3.5-4B')
+hf_config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)
+config_kwargs = hf_to_mcore_config(hf_config)
+config = ModelConfig(
+    params_dtype=torch.bfloat16,
+    tensor_model_parallel_size=TP,
+    pipeline_model_parallel_size=PP,
+    sequence_parallel=True,
+    **config_kwargs)
+
+# 创建模型并加载权重
+mg_models = get_mcore_model(config)
+bridge = config.bridge
+bridge.load_weights(mg_models, model_dir)
+
+# 准备PeftModel并加载LoRA权重
+# 多模态模型建议使用正则表达式指定target_modules
+target_modules = r'^language_model.*\.(in_proj|out_proj|linear_fc1|linear_fc2|linear_qkv|linear_proj)$'
+# 存储成safetensors时，需要存储hf对应的target_modules
+hf_target_modules = r'^model.language_model.*\.(in_proj_qkv|in_proj_z|in_proj_b|in_proj_a|out_proj|gate_proj|up_proj|down_proj|q_proj|k_proj|v_proj|o_proj)$'
+lora_config = LoraConfig(task_type='CAUSAL_LM', r=8, lora_alpha=32, lora_dropout=0.05, target_modules=target_modules)
+peft_models = [get_peft_model(model, lora_config) for model in mg_models]
+# 可选
+# bridge.load_weights(peft_models, model_dir, peft_format=True)
+
+# 导出LoRA权重
+for name, parameter in bridge.export_weights(mg_models, peft_format=True):
+    pass
+
+# 保存LoRA权重
+output_dir = 'Qwen3.5-4B-LoRA'
+bridge.save_weights(mg_models, output_dir, peft_format=True)
+if is_rank0:
+    hf_lora_config = copy.copy(lora_config)
+    hf_lora_config.target_modules = hf_target_modules
+    hf_lora_config.save_pretrained(output_dir)
+```
+
+使用存储下来的LoRA权重：
+```python
+from transformers import Qwen3_5ForConditionalGeneration
+from modelscope import snapshot_download
+from peft import PeftModel
+
+model_dir = snapshot_download('Qwen/Qwen3.5-4B')
+model = Qwen3_5ForConditionalGeneration.from_pretrained(model_dir)
+peft_model = PeftModel.from_pretrained(model, 'Qwen3.5-4B-LoRA')
+```
+
+## ✨ 模型列表
+
+以下为MCore-Bridge支持的模型列表：
+
+| 系列     | model_type                                                   |
+| -------- | ------------------------------------------------------------ |
+| Qwen     | qwen2, qwen2_moe<br />qwen2_vl, qwen2_5_vl, qwen2_5_omni<br />qwen3, qwen3_moe<br />qwen3_vl, qwen3_vl_moe, qwen3_omni_moe<br />qwen3_next, qwen3_5, qwen3_5_moe |
+| DeepSeek | deepseek_v3, deepseek_v32                                    |
+| GLM      | glm4, glm4_moe, glm4_moe_lite<br />glm4v, glm4v_moe, <br />glm_moe_dsa |
+| MiniMax  | minimax_m2                                                   |
+| Kimi     | kimi_k2, kimi_vl                                             |
+| InternLM | internlm3, internvl_chat, internvl                           |
+| Ovis     | ovis2_5                                                      |
+| Llama    | llama, llama4                                                |
+| GPT-OSS  | gpt_oss                                                      |
+| ERNIE    | ernie4_5, ernie4_5_moe                                       |
+| MiMo     | mimo                                                         |
+| Dots     | dots1                                                        |
+| OLMoE    | olmoe                                                        |
 
 
 ## 🏛 License

setup.py

@@ -43,7 +43,7 @@ def parse_requirements(path='requirements.txt'):
         keywords=['transformers', 'LLM', 'lora', 'megatron', 'peft'],
         url='https://github.com/modelscope/mcore-bridge',
         package_dir={'': 'src'},
-        packages=find_packages(include='src'),
+        packages=find_packages('src'),
         python_requires='>=3.8.0',
         classifiers=[
             'Development Status :: 4 - Beta',