SP8192 + PolarExpressNS + MIN_LR + LQER Asym Rank-4 | val_bpb=1.07302 (3-seed mean)

[sahiee-dev]

Results

Seed	val_bpb	artifact_bytes
1337	1.07294	15,955,411
42	1.07270	15,953,179
2025	1.07342	15,951,874
mean	1.07302

All artifacts < 16MB. Train time < 600s. Eval time < 600s.

Technique Stack

Built on PR #1874 (AjAnubolu) — no modifications to the base code.

• SP8192 tokenizer base (@bigbag PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493)
• SmearGate + AttnOutGate width 24 (@MarioPaerle PR RECORD: SmearGate + Attention Output Gate + Legal TTT | val_bpb=1.07139 #1667)
• LoRA-TTT improvements (@renqianluo PR Record: Alpha=144 LoRA + Warm-start A + WD 1.0 — val_bpb 1.07209 (3-seed mean) #1767)
• Phased TTT (@jorge-asenjo PR Add SP8192 Multi-Phase Global SGD + Phased TTT (1.07219 bpb) #1700)
• Polar Express Newton-Schulz coefficients (PR Record: SP4096 + Polar Express + MuonEq-R + Depth Recurrence — 1.0923 BPB (3-seed) #1344)
• MIN_LR=0.10 warmdown floor (PR Record: PR #1736 + Polar Express NS + MIN_LR + Sparse Attn Gate + Fused CE + PR #1767 TTT — val_bpb 1.06335 #1787)
• LQER Asymmetric Rank-4, top-K=3 (PR Record: PR #1787 base + Smear Gate + LQER Asym — val_bpb 1.06157 #1797)

Compliance

• train_under_600s
• artifact_under_16mb
• eval_under_600s
• no_slot
• no_pre_quant_ttt_on_val
• score_first_ttt
• three_seeds (1337, 42, 2025)
• no_text_normalization
• bpb_on_original_bytes

Hardware

8x H100 80GB SXM
PyTorch 2.11.0+cu128

SUBMITTING JUST AS A VALID SUBMISSION WORKING ON BETTER NOVELTY

Note

Submitting as a valid baseline submission. Actively working on novel technique stack for improved val_bpb in follow-up PR.