deepseek-v4 branch: backport #1505 (blockwise-FP8 asserts on public ckpts); BF16-training drift data point

[hvgazula]

Summary

We're reproducing the 8×8 H200 DeepSeek-V4-Flash GRPO profile entirely from public artifacts — the radixark/miles:deepseek-v4 image (digest-pinned), the #1046 overlay (yueming-yuan/miles@032721cd + sgl-project/sglang@b69485ff), and sgl-project/DeepSeek-V4-Flash-FP8 — on Modal (clustered RDMA, no shared NFS). We believe this is the first outside-radixark reproduction. Everything runs end-to-end (checkpoint pipeline, 64-GPU load, SGLang rollout, weight sync, training step); this issue reports two findings from getting there and asks one question.

Finding 1: the deepseek-v4 branch predates the PR #1505 fix — blockwise-FP8 training asserts on public checkpoints

With the profile's --fp8-format e4m3 --fp8-recipe blockwise, the first train step raises:

AssertionError: All-gather requires quantizable tensor for quantizer Float8BlockQuantizer

We traced this to what #1505 (72696ef, merged 2026-06-29) fixes on main — keeping the DSA indexer linear_weights_proj in BF16 via --te-precision-config-file. The pinned deepseek-v4 branch (2026-05-15) predates it. Backporting just the precision-config YAML from #1505 onto the branch resolves the assert (the branch's Megatron already supports te_precision_config_file).

Suggestion: backport #1505 to the deepseek-v4 branch (or note it in the #1046 instructions), since that branch is what the roadmap points public reproducers at.

Finding 2 (data point): pure-BF16 training measures step-0 train/rollout KL ≈ 0.5 on V4-Flash

Before finding #1505 we worked around the assert by dropping the two FP8 flags (BF16 training). Result: train_rollout_kl ≈ 0.506, logprob_abs_diff ≈ 0.69 at step 0 — vs ≈0.02 in the published day-0 numbers — with the R3 replay check (when enabled via --ci-test) reporting 5–20% zero-overlap router top-6 per micro-batch. Everything consistent with FP8-rollout vs BF16-training activation divergence over 43 layers. Presumably why the profile trains blockwise-FP8; sharing since the day-0 write-up's "FP8 rollout + BF16 training" phrasing might lead others down the same path.

Question: is that phrasing intended to describe BF16 master weights with FP8 GEMMs (i.e. the blockwise recipe), rather than pure-BF16 training? A clarifying note would save reproducers a few $290/hr runs.

Minor: tools/fp8_cast_bf16.py on the public FP8 checkpoint

mtp.0.e_proj.weight / mtp.0.h_proj.weight in sgl-project/DeepSeek-V4-Flash-FP8 carry no scale_inv, so the cast passes them through as FP8 ("Warning: Missing scale_inv tensor"), which later breaks torch dist-ckpt serialization (inline_container.cc:664 unexpected pos). Upcasting the two tensors to BF16 before conversion fixes it — a fallback upcast in the cast tool would make the public checkpoint work out of the box.

Happy to share full logs/configs, and to test patches — we have the pipeline cached and can turn runs around quickly.

[hvgazula]

Update with a sharper data point: backporting #1505's precision-config YAML onto the deepseek-v4 branch did not clear the assert — the failing module on our stack is not the indexer but the shared-experts MLP input all-gather under sequence parallelism:

megatron/core/transformer/moe/shared_experts.py:130 -> mlp.py:159
-> transformer_engine/pytorch/module/linear.py:178 forward
-> transformer_engine/pytorch/utils.py:455 assert_dim_for_all_gather
AssertionError: All-gather requires quantizable tensor for quantizer Float8BlockQuantizer

TE 2.10.0 (as baked in the deepseek-v4 image): with --sequence-parallel + --fp8-recipe blockwise, the input activation is quantized for the all-gather, and is_quantizable appears to require block-aligned dims — packed variable-length sequences aren't 128-aligned, so the first shared-experts TELinear asserts. So the question refines to: on which TE/Megatron versions is the 8×8 V4-Flash profile verified? If newer TE gathers-then-quantizes (or pads), the deepseek-v4 image's TE 2.10 may simply be too old for the blockwise recipe with SP, and the branch pin set can't reproduce the profile as published.

[yefei12]

Hello, we internally conducted some RL training and inference experiments based on open-source communities. The training-inference diffs all appear to be quite normal, such as: FP8 training with FP8 inference, FP8 training-inference with QAT, FP8 training-inference with QAT (linear_weights_proj in BF16), and BF16 training-inference.

[RAN-GHR]

Hello, we internally conducted some RL training and inference experiments based on open-source communities. The training-inference diffs all appear to be quite normal, such as: FP8 training with FP8 inference, FP8 training-inference with QAT, FP8 training-inference with QAT (linear_weights_proj in BF16), and BF16 training-inference.

Hi! Thanks for your great work.I wanted to ask whether you ran your evaluation using the configuration in miles/scripts/run_deepseek_v4.py from the repository. If not, would you mind sharing the complete model configuration you used for your experiments (e.g., inference parameters and any other relevant settings)?
We've attached the configuration we used for our evaluation, along with the resulting performance curve, below. Our results differ somewhat from the curve reported by your team, so we'd really appreciate it if you could share your complete evaluation configuration to help us understand the discrepancy.