Record: AttnOutGate + SmearGate + Softcap 15 — val_bpb 1.07750 (3-seed mean)#1880
Open
Meirzhan05 wants to merge 2 commits intoopenai:mainfrom
Open
Record: AttnOutGate + SmearGate + Softcap 15 — val_bpb 1.07750 (3-seed mean)#1880Meirzhan05 wants to merge 2 commits intoopenai:mainfrom
Meirzhan05 wants to merge 2 commits intoopenai:mainfrom
Conversation
3-seed mean: 1.07750 BPB (std 0.0006), all under 16MB. - AttnOutGate (PR openai#1667/openai#1693): per-head data-dependent gate on SDPA output - SmearGate (PR openai#1667 + openai#1851 BOS-fix): forward-1-token residual mixer at embed lane - Lower logit softcap 30 -> 15 (Modded-NanoGPT record openai#18) Stacks cleanly on prior submission (PR openai#1876, 1.08008). Net improvement: -0.00258 BPB.
Meirzhan05
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Record: AttnOutGate + SmearGate + Softcap 15 — val_bpb 1.07750 (3-seed mean)
val_bpb: 1.07750 (3-seed mean, std 0.0006) | ~15.99 MB | 8×H100 SXM
Beats current SOTA (PR #1493, 1.0810) by 0.00350 BPB with std 0.0006 → t-statistic ≈ 5.5, p < 0.001 across 3 seeds. Comparable in magnitude to recent record gaps on the leaderboard (e.g., #2→#1 was 0.0012, #3→#2 was 0.0006).
Three additive zero-cost modifications, all fully precedented and reproducible.
Results (8×H100 80GB SXM, PyTorch 2.9.1+cu128)
Key Changes vs Our Previous Submission (PR #1876, 1.08008 BPB)
Three additive zero-cost modifications:
1. AttnOutGate (PR #1667/#1693)
Per-head data-dependent gate on SDPA output, before
out_proj:W_g: (12 × 8) per layer, zero-init → 2σ(0) = 1 (transparent at init)attn_gateto CONTROL_TENSOR_NAME_PATTERNS)2. SmearGate (PR #1667 + PR #1851 BOS-fix)
Forward-1-token residual mixer at embedding lane:
W: (12 × 1) andλ: scalar — both zero-initinput_ids == BOS_TOKEN_ID(default 1)3. Lower logit softcap 30 → 15 (Modded-NanoGPT record #18)
Single hyperparameter change:
Architecture (unchanged from previous submission)
Training (unchanged)
What We Tried That Did Not Help
Compliance (Issue #1017 conditions)
Condition 1 (Strict Causal Dependence)
Causal attention via
flash_attn_func(causal=True). AttnOutGate uses position-local inputx_t[:12](no leakage). SmearGate is strictly backward-looking (x_{t-1}), with BOS-mask preventing cross-document leakage. TTT only incorporates tokens from already-scored chunks.Condition 2 (Full Normalized Distribution)
F.cross_entropyover full vocab_size logits. Softcap is monotonic (does not mask).Condition 3 (Score-Before-Update)
Each TTT chunk scored under
torch.no_grad()BEFORE any training on it. Model weights at scoring reflect only prior chunks.Condition 4 (Single Left-to-Right Pass)
Single
for ci in range(num_chunks)loop. Each token scored exactly once.Credits