Notable: SP8192 + 3-Layer Recurrence + Parallel Residuals - 5-Seed Quantization Reference and SDClip Ablations

[kiyoaki]

Summary

• 5-seed (42, 314, 999, 1234, 2025) near-frontier reference for the current SP8192 / 3-layer recurrence / parallel residual stack.
  • Post-quant val_bpb mean: 1.08180758 (pop std 0.00052320). Best seed 1234 at 1.08093918. All 5 artifacts clear the 16,000,000 byte decimal cap.
  • Delta vs merged baseline PR Record: SP8192 + GPTQ Embeddings + Depth Recurrence + MuonEq-R + SDClip — val_bpb 1.08563 (5 seed mean) #1394 @clarkkev (1.08563): -0.00382. Near-parity with merged SOTA PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 @bigbag (1.0810).
• Filed as notable non-record. Does not claim a new SOTA technique.
• Packages an empirical pre/post-quant decomposition across all 5 seeds (mean delta post - pre = -0.00545, delta pop std 0.000446), reported as observation rather than causal mechanism claim.
• Includes a negative-result adaptive-clipping ablation sweep (Run 6, 7, 7b, 7c, 8) as a documented reference for future SDClip tuning under the 16 MB cap. Run 7c shows the post - pre sign inverts to +0.01102 BPB when SDClip on the deepest 40 percent of MLP layers is loosened from sigma 12.0 to 14.0.
• Architecture, optimizer, and quantization pipeline are inherited from PR Record: SP8192 + GPTQ Embeddings + Depth Recurrence + MuonEq-R + SDClip — val_bpb 1.08563 (5 seed mean) #1394 @clarkkev. Parallel residuals, 3-layer recurrence, and QK-gain 5.25 are from merged PR lineage (Record: SP8192 + Parallel Residuals + Hessian-Aware SDClip — val_bpb 1.08354 (3-seed mean) #1412, Record: ParallelResiduals + MiniDepthRecurrence, 1.1063 BPB / 1.8679 nats, -0.0072 vs PR #1179, -0.0143 vs merged SOTA #1204, Record: MuonEq-R + 3-Layer Recurrence + WD=0.095 + MLR=0.022 + All-Int6 — val_bpb 1.0900 (3-seed mean) #1331, Record: SP8192 + Parallel Residuals + 3-Layer Recurrence + Token-Only N-gram Tilt — val_bpb 1.08091 (5-seed mean, causal-corrected) #1437, Record: SP8192 + QK-Gain 5 + Legal Score-First TTT — val_bpb 1.08279 (3-seed mean) #1413). Tuning deltas: MATRIX_LR 0.026, WARMDOWN_FRAC 0.75, MUON_WD 0.095, EMA 0.9965.
• Zero TTT. BPB accounting is canonical SP8192 utf-8 byte accounting; unaffected by the SentencePiece byte-accounting bug that prompted the self-closure of PR Record: K_KVShare_Wider full-recipe FLA — val_bpb 1.04090 (3-seed mean) #1687.

Test plan

• Training < 600s on 8xH100 SXM on all 5 seeds
• Eval (sliding-window only, no TTT) < 600s on all 5 seeds
• Artifact < 16,000,000 bytes (decimal) on all 5 seeds; mean 15,989,432 bytes with 7,740 to 13,120 bytes of headroom
• submission.json schema compatible with merged PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 (14 top-level keys, 9 compliance booleans, attribution allow-set)
• Reproduction block in README runs end-to-end from a fresh git clone on 8xH100 SXM