Parallelism - Enable interleaved-pp when layers in pp stage not divisible by vpp degree

[lhzhang333]

Functionality

• Enable interleaved-pp when layers in the first/last pp stage are not divisible by vpp degree with configurable --decoder-first-pipeline-num-layers-split and --decoder-last-pipeline-num-layers-split
  • Auto split: layers-9887, pp-4, vpp-2 ==> [5, 4], [4, 4], [4, 4], [4, 3]
  • Custom split: layers-9887, pp-4, vpp-2, --decoder-first-pipeline-num-layers-split=[4, 5], --decoder-last-pipeline-num-layers-split=[3, 4] ==> [4, 5], [4, 4], [4, 4], [3, 4]

Performance

• Achieve nearly linear combination of the throughput gain of both interleaved-pp and uneven-pp while keeping the same accuracy

[lhzhang333]

lhzhang333 please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree company="AMD"

[Ziyue Yang (yzygitzh)]

Thanks a lot for the PR! I have reviewed an earlier version of the PR and current version looks good to me.
Inside PR description, can we remove all model details and only keep

• A brief functionality description, how to enable it
• Brief description about performance improvement and accuracy conclusion

[lhzhang333]

Thanks a lot for the PR! I have reviewed an earlier version of the PR and current version looks good to me. Inside PR description, can we remove all model details and only keep

• A brief functionality description, how to enable it
• Brief description about performance improvement and accuracy conclusion

Done

We're still working on stabilizing and testing the previous bunch of features, so we decide to delay this PR until next release cycle.

We will also setup a CI to test Megatron-LM itself and all new features. Will be back with comment on how to add a test for this PR.

Really sorry for the back and forth.

[Ziyue Yang (yzygitzh)]

Close and re-open to trigger CI/CD pipeline

[Ziyue Yang (yzygitzh)]

lhzhang333 Please help add test cases for this new feature. An option can be adding uneven VPP cases for both auto split and manual split in tests/unit_tests/transformer/test_transformer_block.py::TestPipelineParallelTransformerBlock.

[github-actions]

Marking as stale. No activity in 60 days.