Skip to content

fix(megatron): finalize async saves on all ranks before hooks#1763

Open
HJSang wants to merge 2 commits into
radixark:mainfrom
GymPod:fix/async-post-save-finalization
Open

fix(megatron): finalize async saves on all ranks before hooks#1763
HJSang wants to merge 2 commits into
radixark:mainfrom
GymPod:fix/async-post-save-finalization

Conversation

@HJSang

@HJSang HJSang commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • finalize Megatron asynchronous checkpoint saves on every training rank before invoking a post-save hook
  • keep post-save hook execution restricted to rank 0
  • add a regression test covering rank 0 and nonzero-rank ordering

Why

Megatron distributed-checkpoint finalization performs distributed coordination. The post-save hook path previously called blocking finalization only on rank 0, while the other ranks returned from save_model. Multi-rank jobs could therefore hang indefinitely before the hook ran.

This preserves the existing contract introduced in #1586: asynchronous saves are fully written before the hook observes them, and the hook itself runs once on rank 0. The only change is that checkpoint finalization now executes collectively.

Validation

  • pytest -q --confcutdir=tests/fast/backends/megatron_utils tests/fast/backends/megatron_utils/test_actor_checkpoint.py (2 passed in radixark/miles:dev)
  • pre-commit run --files miles/backends/megatron_utils/actor.py tests/fast/backends/megatron_utils/test_actor_checkpoint.py
  • multi-rank reproduction: before the fix, rank 0 remained in Megatron async-call finalization after peer ranks returned; collective finalization completed normally with the fix

GPU CI

A maintainer must add run-ci-ckpt and run-ci-megatron before this draft is marked ready for review.

Signed-off-by: Hejian Sang <sanghj0923@gmail.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@HJSang

HJSang commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Can someone take a look at this simple PR? it fixed async bug. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant