Non-record: Pre-Quant AdamW TTT (Compiled) + SP8192 + Depth Recurrence — val_bpb 1.0587 (3-seed mean)

[translatingthename]

Non-record: Pre-Quant AdamW TTT (Compiled) + SP8192 + Depth Recurrence

val_bpb = 1.0587 (3-seed mean, std 0.0004) | ~15.5 MB | 8xH100 SXM

Compliance

This is a non-record submission. The pre-quant TTT implementation (lines 2371-2458 of train_gpt.py) runs 6 AdamW epochs on the full validation token stream before GPTQ quantization, then scores the same tokens via sliding window. This violates Condition 3 of Issue #1017 (score-before-update) and is structurally identical to the pattern in closed PR #1376 and withdrawn PR #1485.

3-Seed Results

Seed	Sliding BPB	Roundtrip BPB	Artifact
42	1.05840	1.06847	15,477,275
1337	1.05856	1.06904	15,439,370
2024	1.05912	1.06921	15,480,770
Mean	1.05869	1.06891	15,465,805
Std	0.00038	0.00037

Contributions

1. Quantifying the Condition 3 boundary

Two measured points bound the illegal TTT contribution at -0.044 BPB (post-EMA 1.103 → post-GPTQ sliding 1.059):

Configuration	BPB	Measured?
Post-EMA (no TTT, no GPTQ)	1.1028	Yes
Post-GPTQ sliding (illegal 6-epoch TTT)	1.0587	Yes
Post-GPTQ sliding (no TTT)	~1.106	Estimated
Post-GPTQ sliding (legal score-first TTT)	~1.104	Estimated from PR #1493 delta

For comparison, the legal score-first TTT in merged PR #1493 contributes approximately -0.002 BPB. This is not an apples-to-apples comparison — different optimizers, epoch counts, chunk sizes, and base models — but the order-of-magnitude gap illustrates why Condition 3 is load-bearing.

On the theoretical ceiling: Issue #1017 states: "Corpus-level TTT has a ceiling of approximately 0.0003 bits" — referring to the gain from closing the train-val distribution gap. However, the author also notes "a model that undertrained on the training distribution can still benefit." Legal TTT's -0.002 exceeds the 0.0003 distribution-gap ceiling because our 600s-capped model is undertrained — this is legitimate undertraining compensation, not memorization. Our illegal TTT's -0.044, however, is 22x larger than legal TTT on a similar architecture, a magnitude not explainable by undertraining compensation alone.

A per-epoch ablation (not performed) would strengthen this argument: linear scaling with epoch count = memorization signature; rapid saturation = generalization.

2. Compiled TTT (2x speedup)

torch.compile(dynamic=False, fullgraph=True) reduces TTT from ~860s to ~426s. Safe in train mode with torch.autocast — no torch.inference_mode() tensor poisoning. Fresh model instance avoids rotary cache contamination. Applies equally to legal score-first TTT.

3. Artifact budget engineering

With SP8192, fitting under 16MB required component-level analysis:

• BigramHash (2048×128): +109KB compressed, ~0.001 BPB at SP8192. Dropped.
• VE dim 128→44: saves 340KB, loses ~0.001 BPB. Optimal via EV analysis (0% pruning risk, 39KB margin).
• Aggressive ±1 pruning destroys quality: 18% pruning = +0.054 BPB. Eliminating pruning via component selection is critical.

Compliance Statement

Violates Condition 3 of Issue #1017. Lines 2371-2458 run 6 AdamW epochs on full val stream before GPTQ. Same tokens scored afterward. No score-before-adapt discipline. Structurally identical to closed PR #1376 and withdrawn PR #1485.

Architecture

11L × 512d × 8H/4KV, MLP 4× (2048), LeakyReLU(0.5)², Partial RoPE (16/64), LN scale, tied embeddings, softcap=30. Depth recurrence L3-5 (14 virtual). Parallel residuals L7+. XSA all layers. VE dim=44 L9-10. SmearGate.

Reproduction

pip install brotli sentencepiece
MATCHED_FINEWEB_REPO_ID=kevclark/parameter-golf python3 data/cached_challenge_fineweb.py --variant sp8192
VOCAB_SIZE=8192 BIGRAM_VOCAB_SIZE=0 VE_DIM=44 TTT_ENABLED=1 SEED=42 \
  torchrun --standalone --nproc_per_node=8 train_gpt.py

Credits

PR #1394 @clarkkev, PR #1493 @bigbag, PR #1485 @ndokutovich, PR #1412 @Robby955, PR #1204 @msisovic, PR #1285 @dexhunter

Checklist

• Folder under records/track_10min_16mb/
• README.md, submission.json, train_gpt.py, 3 seed logs
• All artifacts < 16,000,000 bytes, train < 600s
• Non-record: Condition 3 violation documented

[Bortlesboat]

@translatingthename — compliance flag.

Running our static compliance checker against train_gpt.py, the pre-quant TTT block at lines 2426–2451 trains the model directly on val_tokens for args.ttt_epochs epochs with AdamW.step() before any val scoring is produced:

• line 2426: for epoch in range(args.ttt_epochs):
• line 2429: for start in range(0, total_val_tokens - ttt_seq_len, ...)
• line 2438: chunk = val_tokens[s:s + ttt_seq_len + 1]
• line 2444: loss.backward()
• line 2451: ttt_opt.step()

No preceding no_grad / inference_mode block scores the val tokens before they are used as training targets. This is the exact pattern ruled illegal in the #1376 MatoTeziTanka review and violates Issue #1017 ("the model state used to predict a val token must be determined only by data seen strictly before it").

There is additionally a SLOT-style per-batch delta optimized via AdamW([delta]) against y_batch at lines 2764–2775. In the not args.slot_causal branch (line 2772) the cross-entropy is computed against all val targets without a causal mask, which is the same delta-fitting pattern flagged in #1319 / #1321 / #1313.

Checker: https://github.com/Bortlesboat/parameter-golf-checker