fix: preserve BSHD layout in PPO CP advantages#1752
Open
guapisolo wants to merge 1 commit into
Open
Conversation
PPO GAE dropped qkv_format and per-sample max sequence lengths before CP gather and slicing, so BSHD shards were interpreted with THD offsets. Propagate the layout metadata through advantage computation and add a two-rank regression against a CP=1 baseline.
guapisolo
requested review from
Shi-Dong,
Zhichenzzz,
fzyzcjy,
maocheng23 and
yueming-yuan
as code owners
July 21, 2026 18:30
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Preserve BSHD token ownership while computing PPO advantages with context parallelism.
Symptom & Reproduction
pytest -q -p no:cacheprovider tests/fast/backends/training_utils/test_ppo_cp_advantages.py::test_ppo_bshd_cp_uses_padded_layout_metadataRoot Cause
compute_advantagesdid not forwardqkv_formatormax_seq_lens.get_advantages_and_returns_batchused THD defaults for gather and re-slice.Fix
Propagate
max_seq_lensfrom rollout data and pair it withargs.qkv_formatthroughout PPO GAE. Both CP gather and re-slice now call the existing layout-aware helpers with identical per-sample metadata, while THD and CP=1 keep their previous behavior.Verification
test_ppo_bshd_cp_uses_padded_layout_metadatacompares reconstructed CP=2 BSHD outputs with CP=1 baselines.tests/fast/backends/training_utils/test_ppo_cp_advantages.pypasses all three cases.Review Focus
compute_advantages_and_returnstoget_advantages_and_returns_batch.max_seq_lenspairing in both CP gather and re-slice.