Skip to content

feat: support DeepSeek V4 THD packing#1766

Open
samaritan1998 wants to merge 1 commit into
radixark:mainfrom
samaritan1998:zmm/deepseek-v4-thd-pack-main-0722
Open

feat: support DeepSeek V4 THD packing#1766
samaritan1998 wants to merge 1 commit into
radixark:mainfrom
samaritan1998:zmm/deepseek-v4-thd-pack-main-0722

Conversation

@samaritan1998

Copy link
Copy Markdown

Summary

  • Add true THD packing for DeepSeek-V4 with sample-local cu_seqlens boundaries.
  • Keep CP all-gather, DSV4 window/compressed attention, compressor overlap, RoPE, and TileLang indexer paths boundary-safe.
  • Propagate per-sample padded lengths through response extraction, CP loss masks, log-probs, entropy, and DeepSeek-V4 assistant-only loss masking.
  • Add focused packing, CP-boundary, compressor, indexer workspace, padding-alignment, and loss-mask tests.

Validation

  • LTP job: zmm-dsv4-thd-tp2p8c4-fp32-r13-0722-9e5d50
  • Cluster: book_tjzj_bzz2_hpn1
  • Topology: TP2 / PP8 / CP4 / EP4, 64 GPUs, qkv_format=thd, micro_batch_size=2
  • LTP result: Succeeded
  • Runtime evidence: 7 passed; official torch_dist checkpoint loaded; Final collected 2 samples; step 0 train/loss=0.1832201839, train/grad_norm=1.8685250699; actor train completed normally.
  • Static validation: Ruff, Python compileall, and git diff --check.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant