How long was the training time?

[Yuanhong-Zheng]

Hi, thank for you great work. I noticed that you used 32 H100 in training, and I'm curious about the training time in this compute resource.

[Siddhartha-singh01]

@Yuanhong-Zheng Hello Assign it to me i can resolve this

[wangyu-ustc]

Hi, yes I was using 32 H100 for training and it took 3 days to finish one epoch.

[wtma1999]

Hi, thank you very much for your excellent and solid work.

I am currently attempting to reproduce your training setup and would like to clarify a few details:

Training steps vs. epochs

Using your updated memalpha dataset (~567 samples) and the provided training script, training for 5 epochs results in approximately 85 steps.
However, in the paper’s Implementation Details, you mention training for 205 steps.

Does this mean that you trained for additional epochs, or were there other settings (e.g., different dataset size, sampling strategy, or batch configuration) that contributed to reaching 205 steps?

Training speed and multi-node scaling

I reproduced the Qwen3-4B experiment on 1 node with 8×A100, and the end-to-end training time is quite long (~200 hours). Profiling indicates that a substantial portion of the latency comes from API requests.

I would like to ask:

When using 4 nodes with 32×H100, did you observe a substantial reduction in wall-clock time?

Given that the effective batch size remains 32, my intuition is that scaling across more nodes may not provide a large speedup purely from data parallelism.
Was the improvement mainly due to lower API latency, faster interconnects, or other engineering optimizations?

Any clarification on how the 32×H100 configuration impacts actual training throughput would be greatly appreciated.
Thank you again for your timely responses and for releasing such a strong piece of work.