-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SageMaker] [GraphBolt] Add support for launching GraphBolt jobs on SageMaker #1083
Conversation
91cb60a
to
de02686
Compare
Co-authored-by: xiang song(charlie.song) <[email protected]>
cc85064
to
03bb561
Compare
03bb561
to
99e3b53
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do you launch the convert2graphbolt task?
Co-authored-by: xiang song(charlie.song) <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In overall I think we need to run separate sagemaker regression test before we actually merge the code to make sure there is no other backward compatibility problem.
Co-authored-by: xiang song(charlie.song) <[email protected]>
1b72e36
to
d1f96bb
Compare
Issue #, if available:
Description of changes:
sagemaker/launch_graphbolt_convert.py
will launch the SageMaker job, that downloads the entire partitioned graph to one instance, then runs the GB conversion, one partition at a time. Because DGL writes the new fused CSC graph representation in the same directory as the input data, we can't use one of SageMaker's FastFile modes to stream the data, as that creates read-only filesystems.graphstorm/sagemaker/launch
directory to the runner'sPYTHONPATH
.EDIT: One note about the PR: The changes to the partition launch that use a SageMaker Pipeline are for demonstration purposes, I think I'll remove them alltogether and just have separate partition/gbconvert jobs. But we might want to have an example of how to programmatically build an SM pipeline as an example, e.g. from gsprocessing to training (as SM jobs)
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.