Record val_bpb 1.06996: Independent 3-seed reproduction of PR #1874 + TTT_LORA_RANK=192

[GodlyDonuts]

This submission is fundamentally an independent end-to-end reproduction of PR #1874 by @AjAnubolu, run from scratch on a separate 8×H100 SXM pod across three independent seeds, with one additional hyperparameter change (TTT_LORA_RANK default $128 \rightarrow 192$).

Every numerical claim below maps to a specific line in one of the unedited training logs included in the submission folder; every .int6.ptz artifact described is shipped reload-ready in models/.

Headline numbers (verbatim from the included logs)

Seed	val_bpb (quantized_ttt_phased)	Total submission bytes	Headroom under 16 MB	Eval time (s)
42	1.06927777	15,954,871	45,129	438.3
314	1.07023963	15,954,924	45,076	440.6
999	1.07035739	15,947,796	52,204	434.3
Mean	1.069958	15,952,530	47,470	437.7
Std	0.000592	—	—	—

Comparison:
Vs. current merged SOTA 1.0810 (PR #1493):

• Improvement: 0.011042 nats
• Excess over threshold: 0.006042 nats ($> 0.005$ nat requirement)
• Significance: $t$-statistic = $17.67$ ($df = 2$, one-tailed).
• P-value: Bound $< 0.005$ (Critical $t$ for $p < 0.01$ is $6.965$; for $p < 0.005$ is $9.925$).

Byte-budget compliance

The included train_gpt.py runs _compressed_code_size() at the end of every training run. It reads its own source, runs it through pyminify + lzma + b85, and adds that to the brotli-compressed int6 artifact.

• Minimum Headroom: 45,076 bytes under the 16,000,000-byte cap.
• Code Size: The script on disk is 32,353 B; it self-reports 33,710 B for the budget calculation at runtime.

What's ours, in one diff

- ttt_lora_rank = int(os.environ.get("TTT_LORA_RANK", 128))
+ ttt_lora_rank = int(os.environ.get("TTT_LORA_RANK", 192))

That is the entire code delta vs PR #1874. In our single-seed sweep, rank=192 measured $-0.00019$ nat against rank=128. This is effectively within the noise floor; the primary contribution of this PR is the independent reproduction and the provision of full logs/artifacts.

Compliance with Issue #1017 Track B

• Causality: Sliding-window eval scores from prefix tokens only.
• Score before update: Every chunk is fully scored under torch.no_grad() before any TTT update.
• Single pass: Each token scored exactly once.
• Restricted Techniques: No SLOT, no pre-quant TTT on val data, no ETLB, no n-gram cache.
• Time Constraints: * Train: $< 600$ s on all 3 seeds (9.2 min avg).
  • Eval: $< 600$ s on all 3 seeds (440.6 s max).

Five reload-ready artifacts shipped in models/

Including artifacts allows reviewers to verify the work without a full re-train. (~76 MB total):

• champion_3seed_{42,314,999}.int6.ptz: The headline 3-seed results.
• pr1874_baseline_rank128_seed42.int6.ptz: Baseline reproduction of PR Record: SP8192 + Polar Express NS + MIN_LR + LQER Asym Rank-4 — val_bpb 1.06766 (3-seed mean) #1874.
• sweep_rank192_seed42.int6.ptz: Sweep run for A/B verification.

Regarding PR #1900's provenance review

We are aware of PR #1900 regarding provenance concerns on upstream techniques (MIN_LR and LQER).

1. Independent Verification: Every number here was generated on our own hardware. Logs are provided for trace-ability.
2. Technique Inheritance: We reproduced PR #1874 knowing it uses these high-performance techniques.
3. Policy Compliance: If derivative submissions inherit blocked status, we will not contest closure.
4. Fallback Offer: We can submit a variant with MIN_LR=0.0 and LQER_ENABLED=0 (est. mean: 1.077–1.079 BPB) upon request.

Attribution

• Full Stack: @AjAnubolu (PR #1874)
• Architecture: @dexhunter (PR #1790, #1797) and @clarkkev (PR #1394).
• Frameworks: LQER (PR #1530), Polar Express NS (PR #1667), Score-first TTT (@abaybektursun, PR #549).

---

Submitted by: Saicharan Ramineni (@GodlyDonuts)
Compute: 8×H100 80 GB SXM (RunPod) | PyTorch 2.9.1 | FlashAttention 3