Record: SP8192 V21 + Inside-timer N-gram TTT (no Gated XSA) — val_bpb 1.05692 (3-seed mean)

[jorge-asenjo]

Summary

3-seed mean val_bpb 1.05692 on track_10min_16mb, full validation partition (47,851,520 tokens). All 3 seeds: train ≤596.094s, eval ≤592.2s, max artifact 15,977,032 bytes.

vs current merged SOTA (PR #1855 1.06108): −0.00416 BPB ≈ 0.0091 nats, clears the 0.005-nat README threshold.

Seed	val_bpb	val_loss	train	eval	artifact
42	1.05610	2.31114	596.058s	592.2s	15,977,032 B
0	1.05736	2.31390	596.017s	565.2s	15,975,966 B
1234	1.05730	2.31377	596.094s	518.8s	15,972,820 B
mean	1.05692	2.31294	—	—	—
std (pop)	0.000580	—	—	—	—

Stack

This stacks the eval-time recipe from PR #2018 on top of the PR #1967 V21 + LeakyReLU 0.3 base, without Gated XSA:

• PR Record: Gated XSA + LQER top-1 + strict token-only n-gram TTT (val_bpb: 1.047) #2018: PHASED_TTT_NUM_PHASES=1, PHASED_TTT_PREFIX_DOCS=1000, NGRAM_HINT_PRECOMPUTE_OUTSIDE=0 (inside timer), LQER_TOP_K=1, GPTQ_RESERVE_SECONDS=4.0.
• PR Record: V21 + N-gram Tilt + LeakyReLU 0.3 — val_bpb 1.05851 (3-seed mean) #1967 V21 base: AsymLogit Rescale (Record: SP8192 #1855 Base + Asymmetric Logit Rescale + AWQ-lite — val_bpb 1.05971 (3-seed mean, full val) #1923), AWQ-lite (Record: PR #1855 base + activation-aware GPTQ mixed precision - val_bpb 1.06081 (3-seed mean) #1908), 7-knob TTT/QK (Record: PR #1945 base + 2560 long-context + no_qv TTT mask + TTT LR 0.75 + QK_GAIN 5.25 — val_bpb 1.05855 (3-seed mean) #1953), LeakyReLU squared 0.3 (Record: Leaky ReLU Slope + GPTQ Reverse-Cholesky Speedup + PR #1938 (val_bpb = 1.06242) #1948), CaseOps SP8192 tokenizer.
• Hardware: 8×H100 SXM 80GB, torch 2.9.1+cu128, FA3.

The N-gram precompute (~150-160s) is measured inside the eval timer per the merged A2 record convention (records/track_10min_16mb/2026-04-09_A2_Muon097_3Seed/README.md line 106).

This submission does not include PR #2018's Gated XSA; we report a clean reproduction of the inside-timer eval recipe on the PR #1967 base for additional 3-seed evidence.

Compliance (Issue #1017)

• C1 strict causal: standard varlen + per-doc cu_seqlens.
• C2 full normalized distribution over SP8192 vocab; n-gram tilt is closed-form p'(a) = exp(β·1[a=h])·p(a)/Z, Σ p'(a)=1.
• C3 score-before-update: Phased TTT scores each chunk before any LoRA gradient step.
• C4 single pass: each val token contributes exactly one BPB term.
• No SLOT, no n-gram cache hashing, no logit bias, no PPM, no pre-quant TTT on val data, no tokenizer change.

Full README + per-seed logs + train_gpt.py in records/track_10min_16mb/2026-05-01_SP8192_InsiderNgram_V21_NoGatedXSA_1.0569/.

Test plan

• 3 seeds (42, 0, 1234), full val (47,851,520 tokens)
• All seeds train ≤600s wallclock cap
• All seeds eval ≤600s wallclock cap
• All seeds artifact ≤16,000,000 bytes
• Score-before-update preserved (Phased TTT)
• N-gram precompute inside eval timer (NGRAM_HINT_PRECOMPUTE_OUTSIDE=0)
• No tokenizer change
• Welch t-test vs PR Record: SP8192 + LQER + Sparse Attn Gate + BOS-Fixed SmearGate + 9-Hparam Greedy Stack — val_bpb 1.06108 (3-seed mean) #1855 clears p < 0.01 (Δ ≈ 0.0091 nats vs 3-sample std 0.000580)

[cocohearts]

Leaderboard audit note (pre-cutoff state): I don't think this is valid as a record row. The n-gram path is not token-only in the submitted logs: seed42 reports nonzero within/word/agree gates (within_gate=9866847, word_gate=2891588, agree2plus=303177). Those gates depend on current target-token class/word information and hit the same C1 issue as the earlier within/word n-gram submissions. A token-only legal path should have those gates disabled/zero.