Record: Improved Parallel Residuals + Systems Optimization — val_bpb 1.0752 (3-seed mean)

[codemath3000]

Summary

• val_bpb: 1.0752 (3-seed mean, std 0.0006) | 8xH100 SXM, 600s | Legal TTT
• Systems-level optimizations on PR Record: ImprovedParallelResiduals, 1.0758 BPB / 2.7789 nats, -0.0020 BPB / -0.0052 nats vs PR #1523 #1529's dual-lane parallel residual architecture: fused Muon kernel, batched EMA, loader prealloc
• Identical ML; faster step time yields ~20 extra training steps in the same 600s budget
• Per Record Criterion 1: "For submissions that improve speed through systems optimization without changing the ML, this requirement [0.005 nats] is waived." This submission changes only systems-level code (kernel fusion, batched ops, memory preallocation) without altering model architecture, optimizer logic, loss function, or any hyperparameter, meaning the 0.005 nats threshold is waived. Additionally, relative to the current official leaderboard SOTA at the time of writing (PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493), this submission wins by 0.0150 nats, well above the threshold.

Submission series: This PR is one of three related submissions applying the same systems optimizations to different base stacks (PR #1493, PR #1529, PR #1578). We submit against multiple bases so that a ready-to-merge option exists regardless of how the pending PRs are resolved. Judges should feel free to evaluate whichever base(s) they consider valid and disregard the rest.

Results

Seed	TTT BPB	Artifact
1337	1.0745	15,983,819
2024	1.0755	15,982,374
42	1.0755	15,979,637
Mean	1.0752	15,981,943

CUTLASS EVT Build

Required for full throughput. Source included in cutlass_evt_fusion/:

git clone https://github.com/NVIDIA/cutlass.git /opt/cutlass
cd /opt/cutlass && git checkout 08185b9c3e90510ee2b656662ed0d53b06d28157
pip install --no-build-isolation ./cutlass_evt_fusion

Test plan

• 3-seed training on 8xH100 SXM (seeds 1337, 2024, 42)
• All artifacts under 16MB
• All runs under 600s training + 600s eval
• Round-trip quantization + TTT verified
• Judges verify reproducibility

🤖 Generated with Claude Code