Record: PR #2014 stack + LeakyReLU 0.3 + strict in-timer n-gram TTT (val_bpb 1.0560)

[simon-marcus]

Record: PR #2014 stack + LeakyReLU 0.3 + strict in-timer n-gram TTT (val_bpb 1.05601)

3-seed mean: val_bpb 1.05601155 | max 15,997,965 bytes | 8xH100 SXM | 600s train + in-timer eval

Results

Seed	Train steps	Pre-quant val_bpb	Quantized val_bpb	Post-TTT val_bpb	Train time	Eval time	Artifact bytes	Notes
42	4,875	1.05971819	1.06768136	1.05528133	596.1s	580.7s	15,997,965	in-timer n-gram hints, prefix=2500, chunk=64
314	4,879	1.06052597	1.06877627	1.05629015	596.0s	583.1s	15,992,681	in-timer n-gram hints, prefix=2500, chunk=64
0	4,877	1.06101308	1.06891132	1.05646316	596.1s	545.2s	15,996,288	in-timer n-gram hints, prefix=2500, chunk=64
Mean	4877	1.06041908	1.06845632	1.05601155	596.1s	569.7s	15,997,965 max	3 seeds

Compared with the last merged leaderboard record (#1855, 1.06107587 BPB), this 3-seed mean improves val_bpb by 0.00506432.

Summary

This submission starts from the PR #2014 strict-compliance stack and adds two changes:

1. LeakyReLU-square slope 0.3. PR Record: PR1855/PR1953 base + Progressive context growth (val_bpb: 1.05759, 3-seed) #2014 inherited the older LeakyReLU(0.5)^2 MLP slope. This changes the fused/eager LeakyReLU-square path to slope 0.3, following the later PR Record: V21 + N-gram Tilt + LeakyReLU 0.3 — val_bpb 1.05851 (3-seed mean) #1967 lineage.
2. Strict in-timer online n-gram tilt during TTT eval. This ports the in-timer online n-gram tilt approach we introduced in PR Record: Gated XSA + LQER top-1 + strict token-only n-gram TTT (val_bpb: 1.047) #2018 into the PR Record: PR1855/PR1953 base + Progressive context growth (val_bpb: 1.05759, 3-seed) #2014 progressive-context / short-doc TTT path. Hints are built causally from validation tokens inside the measured TTT eval timer (NGRAM_HINT_PRECOMPUTE_OUTSIDE=0) and applied as a scoring-time posterior adjustment to per-token NLL.

The n-gram path does not add model parameters and has no artifact-size cost beyond source files. The run keeps the PR #2014 global prefix phase (PHASED_TTT_PREFIX_DOCS=2500) and uses larger TTT chunks to fit hint construction and scoring inside the 600s eval budget.

What changed vs PR #2014

Component	PR #2014	This submission
Base stack	Progressive 3k context growth + ShortDoc TTT + CaseOps + LQER + AWQ-lite	Same
LeakyReLU-square slope	0.5	0.3
Eval-time n-gram tilt	off	on, causal, in timer
N-gram hint timing	n/a	NGRAM_HINT_PRECOMPUTE_OUTSIDE=0
Global phased TTT prefix	2500 docs	2500 docs
TTT chunking	48 / short 24	64 / short 32
Artifact	PR #2014 per-group compressed artifact	same compression path, 15,997,965 bytes max

Compliance notes

• Training cap: all three seeds stopped under 600s (596.142s, 596.003s, 596.061s).
• Eval cap: all three final TTT evals are under 600s (580.667s, 583.138s, 545.161s). All use NGRAM_HINT_PRECOMPUTE_OUTSIDE=0, so n-gram hint generation is inside the measured eval timer.
• Artifact cap: max observed Total submission size quantized+pergroup is 15,997,965 bytes, under 16 MB.
• Score-first TTT: the LoRA TTT path scores each chunk before any per-doc update. The global prefix SGD phase runs after the prefix docs have already been scored.
• N-gram causality: hints are generated by a single left-to-right pass over validation tokens and aligned to target positions. The tilt uses prefix-derived hint IDs and boosts; it does not inspect future tokens for the scored position.

Key settings

CASEOPS_ENABLED=1
VOCAB_SIZE=8192
TRAIN_SEQ_LEN=3072
ROPE_TRAIN_SEQ_LEN=3072
[email protected],[email protected],[email protected]
TRAIN_SEQ_SCHEDULE_MODE=wallclock
SEQ_CHANGE_WARMUP_STEPS=32
COMPILE_SHAPE_WARMUP=1
EVAL_SEQ_LEN=3072
EVAL_STRIDE=1536

LEAKY_RELU_SQ_SLOPE=0.3

TTT_ENABLED=1
TTT_EVAL_SEQ_LEN=3072
TTT_BATCH_SIZE=24
TTT_CHUNK_SIZE=64
TTT_SHORT_SCORE_FIRST_ENABLED=1
TTT_SHORT_DOC_LEN=2000
TTT_SHORT_CHUNK_SIZE=32
TTT_SHORT_SCORE_FIRST_STEPS=256:16,2000:32
TTT_LORA_RANK=80
TTT_LORA_LR=0.0001
TTT_LOCAL_LR_MULT=0.75
TTT_MASK=no_qv
TTT_Q_LORA=0
TTT_V_LORA=0
TTT_WEIGHT_DECAY=0.5
TTT_BETA2=0.99
PHASED_TTT_PREFIX_DOCS=2500
PHASED_TTT_NUM_PHASES=1

NGRAM_TILT_ENABLED=1
NGRAM_HINT_PRECOMPUTE_OUTSIDE=0
TOKEN_ORDER=16
TOKEN_THRESHOLD=0.800
TOKEN_BOOST=2.625
WITHIN_TAU=0.450
WITHIN_BOOST=0.750
WORD_ORDER=4
WORD_NORMALIZE=strip_punct_lower
WORD_TAU=0.650
WORD_BOOST=0.750
AGREE_ADD_BOOST=0.500

WARMDOWN_FRAC=0.85
BETA2=0.99
QK_GAIN_INIT=5.25
SPARSE_ATTN_GATE_ENABLED=1
SPARSE_ATTN_GATE_SCALE=0.5
GATED_ATTN_QUANT_GATE=1
SMEAR_GATE_ENABLED=1
GATE_WINDOW=12
FUSED_CE_ENABLED=1
MATRIX_LR=0.026
MIN_LR=0.1
GRAD_CLIP_NORM=0.3
EMBED_BITS=7
EMBED_CLIP_SIGMAS=14.0
MATRIX_CLIP_SIGMAS=12.85
ATTN_CLIP_SIGMAS=13.0
MLP_CLIP_SIGMAS=11.5
LQER_ENABLED=1
LQER_RANK=4
LQER_TOP_K=3
LQER_FACTOR_BITS=4
LQER_ASYM_ENABLED=1
LQER_ASYM_GROUP=64
AWQ_LITE_ENABLED=1
AWQ_LITE_BITS=8
AWQ_LITE_GROUP_TOP_K=1
AWQ_LITE_GROUP_SIZE=64
ASYM_LOGIT_RESCALE=1
COMPRESSOR=pergroup
GPTQ_RESERVE_SECONDS=4.0
GPTQ_CALIBRATION_BATCHES=16
VAL_LOSS_EVERY=0

Files

• train_gpt.py — full script for the candidate.
• online_ngram_tilt.py, online_ngram_state.c — online causal n-gram hint builder and scoring-time tilt helper, from the PR Record: Gated XSA + LQER top-1 + strict token-only n-gram TTT (val_bpb: 1.047) #2018 in-timer n-gram tilt work.
• train_seed42.log — seed-42 training + quantization log for the artifact reused by the eval sweep.
• eval_seed42_ngram_p0_c64.log — earlier seed-42 in-timer n-gram TTT eval log with prefix disabled.
• eval_seed42_ngram_p2500_c64.log — seed-42 frozen-settings in-timer n-gram TTT eval log.
• train_eval_seed314.log — seed-314 training, quantization, and in-timer TTT eval log.
• train_eval_seed0.log — seed-0 training, quantization, and in-timer TTT eval log.
• prepare_caseops_data.py, lossless_caps.py, tokenizers/...model — CaseOps data/tokenizer helpers from the merged Record: SP8192 + LQER + Sparse Attn Gate + BOS-Fixed SmearGate + 9-Hparam Greedy Stack — val_bpb 1.06108 (3-seed mean) #1855 lineage.
• submission.json — structured 3-seed metadata.

Reproducing

After preparing the CaseOps data and tokenizer, run with the environment above:

SEED=42 DATA_PATH=./data/datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved TOKENIZER_PATH=./data/tokenizers/fineweb_8192_bpe_lossless_caps_caseops_v1_reserved.model torchrun --standalone --nproc_per_node=8 train_gpt.py

For the eval-only sweep used here, load the saved quantized artifact and run with:

TTT_EVAL_ONLY=1 NGRAM_TILT_ENABLED=1 NGRAM_HINT_PRECOMPUTE_OUTSIDE=0 PHASED_TTT_PREFIX_DOCS=2500 TTT_CHUNK_SIZE=64 TTT_SHORT_CHUNK_SIZE=32 TTT_SHORT_SCORE_FIRST_STEPS=256:16,2000:32 torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

This is a stack on top of the recent strict-compliance CaseOps line. Most directly:

• PR Record: PR1855/PR1953 base + Progressive context growth (val_bpb: 1.05759, 3-seed) #2014 by @simonbissonnette — progressive 3k context growth and short-doc TTT base.
• PR Record: Gated XSA + LQER top-1 + strict token-only n-gram TTT (val_bpb: 1.047) #2018 — in-timer online n-gram tilt during TTT eval.
• PR Record: V21 + N-gram Tilt + LeakyReLU 0.3 — val_bpb 1.05851 (3-seed mean) #1967 / PR Record: Leaky ReLU Slope + GPTQ Reverse-Cholesky Speedup + PR #1938 (val_bpb = 1.06242) #1948 lineage — LeakyReLU-square slope 0.3 evidence.
• PR Record: SP8192 + LQER + Sparse Attn Gate + BOS-Fixed SmearGate + 9-Hparam Greedy Stack — val_bpb 1.06108 (3-seed mean) #1855 by @codemath3000 — merged CaseOps / SparseGate / LQER / per-group compression lineage and data-prep precedent.
• PR Record: PR #1736 + Polar Express NS + MIN_LR + Sparse Attn Gate + Fused CE + PR #1767 TTT — val_bpb 1.06335 #1787, Record: SP8192 + CaseOps + GatedAttn + QuantGate + Loop45 + PhasedTTT — val_bpb 1.06549 #1736, Record: CaseOps Tokenizer + Tapered WD - val_bpb 1.0678 (3-seed mean) #1729, RECORD: SmearGate + Attention Output Gate + Legal TTT | val_bpb=1.07139 #1667, Record: VarLen Attention + Fused MLP + Multi-Phase Global SGD TTT — val_bpb 1.07193 (3-seed mean) #1626, Record: Varlen attention + fused MLP + doc-independent TTT (1.07336) #1530, Record: SP4096 + Polar Express + MuonEq-R + Depth Recurrence — 1.0923 BPB (3-seed) #1344, Record: 11L EMA + Int6 + XSA + LeakyReLU² + Partial RoPE (val_bpb: 1.1309) #493, New SOTA: 1.12676 BPB - 11L XSA-all(11) + GPTQ-lite + EMA + Late QAT #478, Record: 11L Partial RoPE + LN Scale + EMA + XSA4 (val_bpb: 1.1248) #315, and SmearGate + BigramHash + Int6 + SWA + U-Net Skips (1.1518 BPB) #289 for the underlying architecture, optimizer, tokenizer, quantization, compression, and legal score-first TTT components.

[codemath3000]

Flagging what looks like a Condition 1 compliance issue.

This PR uses the n-gram tilt's within-word and word-start channels (WITHIN_BOOST=0.750, WORD_BOOST=0.750, AGREE_ADD_BOOST=0.500). These channels were identified as a Rule 1 violation in review of PR #1420, and the author accepted the finding and removed them. Quoting PR #1420's own body:

"@Gusanidas identified that within_hint and word_hint used is_bnd/is_ws flags derived from tokens_[p] (the target token) to gate whether a hint was produced — a Rule 1 violation."

"The gating decision 'should I produce a hint at this position?' depended on whether the target token was a word boundary or had a leading space. This meant the probability distribution P(x_t | x_1...x_{t-1}) changed depending on the value of x_t itself."

"Conclusion: The within/word channels' -0.0025 BPB contribution came entirely from target-dependent gating. Without it, they add noise. Only token_hint (orders 8–16) produces a legitimate improvement."

The merged precedent for this exact n-gram code (PR #1514, merged 2026-04-29) explicitly excludes both channels:

"Causal n-gram tilt — only the prefix-only token expert is active. The within-word and word-start experts from PR #1420 are explicitly zeroed out."

The same target-dependent gating is still present in this PR's code:

• online_ngram_state.c:321-324, 351, 361: at scoring position i, the C kernel reads tok = tokens[i] and gates the within-channel hint emission on boundary_lut[tok] and starts_new_word_lut[tok] — both functions of the realized target.
• online_ngram_tilt.py:181-188: WordStartState gates word-channel hint emission on the same target-derived flags.

With WITHIN_BOOST and WORD_BOOST nonzero, the tilt at position t depends on whether x_t is a boundary/word-start token — exactly the pattern the merged ruling excluded.

The README's compliance section addresses future-token leakage ("does not inspect future tokens") but not the target-token-at-position-t issue that PR #1420's review identified. PR #2018 is cited as additional precedent but is currently OPEN/unmerged; only PR #1514 is binding precedent on this code, and it disabled these channels.

Could the author and maintainers take a look? Happy to be corrected if I've misread something.