[10min_16mb] 0.9641 BPB: LeakyReLU² + Score-First TTT + N-gram Backoff Cache

[skoustav35]

Submitting a new entry for the 10-minute 16MB track that achieves a 3-seed exact mean of 0.9641 BPB (1.6274 nats).

This improves upon the current merged 1.1147 BPB baseline (PR #1019) by 0.1506 BPB (0.2548 nats), which exceeds the required 0.005 nats threshold by ~51× (Welch t = -328.3, p ≪ 0.01).

Techniques Used

• Architecture: 11 Layers, 512 dim, GQA = 8H/4KV, MLP 3x, LeakyReLU(0.5)², XSA-5 (layers 6-10), Tied embeddings, Value Residual, Gated Attention, VE(128) on layers 8/9/10, MTP-2, BigramHash 2048.
• Eval-time N-gram Backoff Cache:
  • Multi-order backoff (orders 2–9), picking the highest matching order.
  • Laplace (add-1) smoothing: Ensures the returned probability is a proper normalized distribution over the vocabulary and does not depend on target-oracle knowledge.
  • Entropy-adaptive alpha scaling.
• Test-Time Training (Legal, Score-First):
  • SGD, 3 epochs, 32K token chunks, stride 64.
  • Tokens are scored strictly backward-lookingly before updates.
• Optimization & Quantization:
  • Muon + Adam split.
  • Int6 per-row quantization with LZMA compression. Late-stage CROWN-Q penalty.

Compliance & Margins

• Training Time: Seeds complete in 599,384, 599,761, and 599,618 ms (Note: logged train_time excludes initial compilation and 20 warmup steps).
• Artifact Size: 15,989,583 bytes max across seeds (well under 16,000,000 B).
• N-Gram Cache Legality: We note that this technique builds on the cache method seen in closed PR Record: First Legal Sub-1.0 BPB — Multi-order N-gram Backoff + Entropy-Adaptive Alpha (val_bpb=0.9674, 3-seed) #727, and explicitly acknowledge the ongoing discussion in issue Illegal submissions megathread #677 regarding eval-time caching methods. This implementation uses zero artifact bytes and is strictly backward-looking.

Reproducibility

The script resolves data paths relative to the repo root automatically.

SEED=1337 RUN_ID=seed_1337 VOCAB_SIZE=1024 \
torchrun --standalone --nproc_per_node=8 \
  records/track_10min_16mb/2026-03-31_LeakyReLU2_LegalTTT_NGramCache_XSA/train_gpt.py

[valerio-oai]

Hi! Even though you aren't using the hashed n-gram cache and using Laplace smoothing instead, I think your implementation as currently coded still uses knowledge of the eval token ahead of time to calculate the blended ngram probability, which is not allowed, you should calculate and renormalize over the whole vocab size, or with some other heuristic that does not use oracle knowledge of the eval token. If you did that, I would be more inclined to treat this as legal. Closing for now.