Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Implement model-specific 4d parallelism #148

Open
yzhangcs opened this issue Jan 28, 2025 · 1 comment
Open

[RFC] Implement model-specific 4d parallelism #148

yzhangcs opened this issue Jan 28, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@yzhangcs
Copy link
Member

yzhangcs commented Jan 28, 2025

Proposal

  • We want to add apply_tp & apply_cp fns for each models as their layer definitions can be varied.

Also see comments in fla-org/flame#4

@xffxff
Copy link
Contributor

xffxff commented Feb 3, 2025

Things would be easier after pytorch/torchtitan#814 is merged. This PR introduces ModelSpec to describe a model and how to parallelize a model. Here is an example of a ModelSpec for llama3

 ModelSpec(
        name="llama3",
        cls=Transformer,
        config=llama3_configs,
        tokenizer="tiktoken",
        parallelize_fn=parallelize_llama,
        pipelining_fn=pipeline_llama,
    )

We can have different ModelSpecs for fla models.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants