Cautious Muon + SP4096 + Depth Recurrence — val_bpb 1.1604 (non-record)

[X-Abhishek-X]

Summary

• val_bpb = 1.1604 (3-seed mean, std = 0.0033)
• Applies Cautious Muon (arXiv:2411.16085) to the Muon optimizer — masks Newton-Schulz updates where the orthogonalized direction disagrees with the raw gradient sign
• Built on PR Record: SP4096 + Depth Recurrence + Parallel Residuals + MuonEq-R + QK-Gain 5.0 — val_bpb 1.0897 (3-seed mean) #1334 (@aryanbhosale) base: SP4096, depth recurrence, parallel residuals, MuonEq-R, QK-Gain 5.0
• Track A fixed predictor — no TTT, no SLOT, no eval-time adaptation

Per-Seed Results

Seed	val_bpb	val_loss	Artifact Size
42	1.1568	2.6619	15,179,504 B
314	1.1611	2.6717	15,173,470 B
999	1.1634	2.6770	15,159,223 B
Mean	1.1604	2.6702	15,170,732 B

Cautious Muon (Key Modification)

After Newton-Schulz orthogonalization and MuonEq-R row normalization, the update is masked:

caution_mask = (g * raw_grad > 0).to(g.dtype)
g = g * caution_mask / caution_mask.mean().clamp_min(1e-3)

This filters stale momentum directions, providing ~1.47x effective convergence per step with zero parameter overhead.

Architecture Stack

• SP4096 BPE tokenizer (PR Record: 4096-Vocab + 4.0-MLP-mult + 0.085-WD + Simplifications — val_bpb 1.09785 (3-seed mean) #1218, @clarkkev)
• 11 physical layers → 13 virtual (depth recurrence layers 4,5 from step 3000)
• Parallel residuals from layer 7 (separate attn/MLP lanes)
• MuonEq-R row normalization (arXiv:2603.28254)
• QK-Gain 5.0, EMA 0.997
• Full GPTQ INT6 + Brotli compression
• Non-matrix params use standard AdamW

Compliance

• Track A fixed predictor
• Causal, normalized via softmax (F.cross_entropy)
• Artifact under 16MB (max 15,179,504 bytes)
• Training under 600s wallclock on 8xH100 SXM

Reproduction

SEED=42 torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

• PR Record: SP4096 + Depth Recurrence + Parallel Residuals + MuonEq-R + QK-Gain 5.0 — val_bpb 1.0897 (3-seed mean) #1334 (@aryanbhosale) — base architecture
• PR Record: 4096-Vocab + 4.0-MLP-mult + 0.085-WD + Simplifications — val_bpb 1.09785 (3-seed mean) #1218 (@clarkkev) — SP4096 tokenizer
• Liang et al. (arXiv:2411.16085) — Cautious Optimizers

[Copilot]

Pull request overview

Adds a new Track A (10min_16mb) records folder documenting an experiment that applies “Cautious Muon” masking to Muon optimizer updates on top of the existing SP4096 + depth recurrence + parallel residuals stack, and includes 3-seed training logs plus submission metadata.

Changes:

• Adds a new record folder with README.md describing the technique and results.
• Adds submission.json capturing aggregated metrics and per-seed artifact sizes.
• Adds per-seed training logs and a self-contained train_gpt.py entrypoint.

Reviewed changes

Copilot reviewed 3 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/README.md	Documents the method (Cautious Muon) and experiment results + reproduction instructions.
records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/submission.json	Provides submission metadata and aggregated/per-seed metrics.
records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_gpt.py	Self-contained training script payload for reproducing the run.
records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_seed42.log	Seed 42 training + eval log for the submission.
records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_seed314.log	Seed 314 training + eval log for the submission.
records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_seed999.log	Seed 999 training + eval log for the submission.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The results table rows start with double pipes (|| ...), which GitHub Markdown renders as an extra empty first column. Use single leading | for each row so the table displays correctly.

[Copilot]

The description of Cautious Muon says the mask is applied “after Newton–Schulz orthogonalization and MuonEq-R row normalization”, but later the stack list claims MuonEq-R happens before Newton–Schulz. Please make the ordering consistent with the actual implementation to avoid confusion when reproducing.

[Copilot]

This bullet claims “MuonEq-R row normalization before Newton-Schulz”, which contradicts the earlier description of the masking point. Please align the ordering here with the actual optimizer pipeline (and with the earlier section) so readers don’t implement the wrong sequence.

Suggested change

	- **MuonEq-R** row normalization before Newton-Schulz (arXiv:2603.28254)
	- **MuonEq-R** row normalization after Newton-Schulz orthogonalization (arXiv:2603.28254)

[Copilot]

Most existing 10min_16mb submission.json files include a bytes_total field (e.g., records/track_10min_16mb/2026-03-22_11L_EMA_GPTQ-lite_warmdown3500_QAT015_1.1233/submission.json:9). This submission uses artifact_bytes_* instead; consider adding bytes_total (and optionally bytes_code) for consistency and easier leaderboard/tooling ingestion.

[Copilot]

The date field is just YYYY-MM-DD, while other submissions typically use an ISO-8601 timestamp (e.g. 2026-03-22T00:00:00Z). Using a consistent timestamp format helps automated consumers parse and sort submissions reliably.

Suggested change

	"date": "2026-04-05",
	"date": "2026-04-05T00:00:00Z",

[Copilot]

train_gpt.py is an opaque exec(lzma_decompress(base85(...))) blob. This makes review and reproducibility auditing difficult (and can trip security scanners). If possible, include the actual Python source (even if minified) or at least provide the decompressed source alongside this file so reviewers can inspect the implementation of the claimed optimizer change.

[Copilot]

This submission is labeled as a “Record” (folder/PR title), but the reported mean val_bpb = 1.1604 is substantially worse than the current 10min_16mb leaderboard entries (e.g. 1.1228 in the repo README). Consider renaming the PR/folder/README name to avoid implying it’s a new SOTA record if it’s intended as a non-record/ablation submission.

[MatoTeziTanka]

Community Review — Cautious Muon + SP4096 + Depth Recurrence — val_bpb 1.1604 (non-record)

BPB: 1.1604 | Compliance: LOOKS CLEAN — score-first-per-chunk TTT (legal #1416/#1423 pattern)

What I found in the code (head SHA a6070d3e54dd, file records/track_10min_16mb/2026-04-05_CautiousMuon_SP4096_DepthRecur/train_gpt.py):

The TTT path at line 1544 implements the score-first-per-chunk pattern: each chunk is scored under torch.no_grad() / inference_mode() before the base_model.train() + SGD adaptation runs on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. This is the structural shape the legal frontier uses (PRs #1416 erichroepke, #1423 aryanbhosale).

Per Issue #402 and Issue #677, TTT is legal when each token is scored before the adapter updates on it, and that's what the code does here — chunk ci is scored under weights adapted only on chunks 0..ci-1. No prequant_ttt_adapt_adamw(val_tokens, ...) multi-epoch fine-tune, no scored-region SLOT, no target-in-key n-gram cache.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 4.56s, dim=512, layers=11, vocab=4096, code=24659 B, SMOKE_TEST_PASS

Verdict: LOOKS CLEAN.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: MERGE pending standard checks (3-seed validation, 16MB artifact cap, 10-min wallclock on 8×H100 SXM). The compliance picture matches the legal reference frontier and no flags were raised by the classification pass.

Auto-classification caveat: this review was drafted by the AST-based classifier against a template derived from manually-reviewed cluster PRs (#1420, #1450, #1487, #1541, #1529, #1533, #1518). If I've misread a subtlety in your eval path — e.g., multi-epoch TTT that I mistook for single-pass, or a target-in-key lookup I missed in a helper function — please flag it and I'll re-run the audit manually.

---

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 4.56s, dim=512, layers=11, vocab=4096, code=24659 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.