Record: SP8192 #1855 Base + Asymmetric Logit Rescale + AWQ-lite — val_bpb 1.05971 (3-seed mean, full val)

[jorge-asenjo]

Summary

Verbatim PR #1855 stack with one orthogonal eval-path addition: asymmetric logit rescale (modded-nanogpt @classiclarryd PR #181). Two learnable scalars (softcap_pos, softcap_neg) replace the single logit_softcap in forward_logits and forward_ttt, trained inside Phased TTT via the global SGD loop. Init at logit_softcap=30.0 so eval is identity at start; train-time fused-CE kernel left untouched.

3-seed Results

Seed	val_bpb	artifact
42	1.06533	15,902,200 B
0	1.06643	15,899,693 B
1234	1.06554	15,907,523 B
mean	1.06577	15,903,139 B avg

Reproduced on 8× H100 80GB SXM, torch 2.9.1+cu128, FA3, fused softcapped CE, lrzip pergroup compression.

What changed vs #1855

In class GPT:

• New init flag asym_logit_enabled gated by ASYM_LOGIT_RESCALE env var
• Two nn.Parameter scalars softcap_pos, softcap_neg (init 30.0, fp32)
• Helper _apply_asym_softcap(logits) doing where(logits>0, sp*tanh(logits/sp), sn*tanh(logits/sn))
• forward_logits and forward_ttt call the helper when flag is on; training path unchanged

The two new scalars land in the passthrough float16 list at serialize time (8 bytes total, lossless under per-group lrzip).

Test plan

• Reproduce SEED=42 → expect within ±0.001 of 1.06533
• Verify training path numerics unchanged (fused CE kernel gets pre-softcap logits unchanged)
• Compliance: score-first TTT preserved, full normalized distribution over Σ, no rescoring, single L→R pass over validation

Attribution

• @codemath3000 — PR Record: SP8192 + LQER + Sparse Attn Gate + BOS-Fixed SmearGate + 9-Hparam Greedy Stack — val_bpb 1.06108 (3-seed mean) #1855 base stack
• @dexhunter — PR Record: PR #1787 base + Smear Gate + LQER Asym — val_bpb 1.06157 #1797 (LQER asym + SmearGate base) and PR Record: SP8192 + CaseOps + GatedAttn + QuantGate + Loop45 + PhasedTTT — val_bpb 1.06549 #1736 (CaseOps + GatedAttn)
• @nprime06 — PR Record: PR #1736 + Polar Express NS + MIN_LR + Sparse Attn Gate + Fused CE + PR #1767 TTT — val_bpb 1.06335 #1787 (Polar Express NS + MIN_LR + Sparse Attn Gate + Fused CE)
• @MarioPaerle — PR RECORD: SmearGate + Attention Output Gate + Legal TTT | val_bpb=1.07139 #1667 (SmearGate origin)
• @renqianluo — PR Record: Alpha=144 LoRA + Warm-start A + WD 1.0 — val_bpb 1.07209 (3-seed mean) #1767 (LoRA TTT improvements)
• @classiclarryd — modded-nanogpt PR 1.1190 BPB — 11L LeakyReLU² XSA4 PartialRoPE LNScale EMA ParallelMuon TTT #181 (Asymmetric Logit Rescale, the orthogonal addition over Record: SP8192 + LQER + Sparse Attn Gate + BOS-Fixed SmearGate + 9-Hparam Greedy Stack — val_bpb 1.06108 (3-seed mean) #1855)
• Phased Multi-Phase Global SGD TTT retained from PR Add SP8192 Multi-Phase Global SGD + Phased TTT (1.07219 bpb) #1700.

[h1beee]

PR #1855 reports 1.061 bpb, but yours is higher at 1.065 bpb?

[jorge-asenjo]

The +0.0047 gap is an environment offset, not a regression. I reproduced the #1855 stack verbatim (same code, same env vars, same seeds) on my pod and got 1.06754 (seed 42) — i.e. the verbatim #1855 baseline measures ~+0.006 higher under my setup than its reported 1.06108. AsymLogit on top of that same baseline lands at 1.06577 (3-seed mean), so the delta on this PR is −0.00177.

What's submitted here is the AsymLogit delta on top of the #1855 stack, not a claim against the #1855 absolute number. A third party reproducing #1855 verbatim and then toggling ASYM_LOGIT_RESCALE=1 should see the same sign.

[codemath3000]

Thanks for putting up the repro detail @jorge-asenjo. I took a look at the seed-42 logs side-by-side, and the gap looks like it's a different validation set rather than an environment offset:

metric	this PR seed-42	#1855 seed-42
val_tokens	9,662,464	47,851,520
train_shards	1502	80
pre-quant post-EMA val_bpb (no TTT, no AsymLogit)	1.07019	1.06396
post-quant val_bpb (no TTT)	1.07896	1.07254
final TTT-phased val_bpb	1.06533	1.05989

All hyperparameters in the printed Hyperparameters: block are byte-identical between the two runs. The +0.0062 gap is already present at the pre-quantization, post-EMA diagnostic — i.e. before TTT, GPTQ, pergroup serialization, or any AsymLogit code runs. That's a deterministic forward pass, so the only thing it can be measuring is the val corpus itself.

Your val_tokens is 9,662,464; the standard FineWeb10B SP8192 caseops val set used by #1855 (and the prior records in this stack) has 47,851,520 tokens — about 5× larger. With val_doc_fraction: 1.0 in your log, that points to a smaller val partition on disk rather than a sampling difference.

The intra-environment delta you're showing (verbatim #1855 = 1.06754, this PR = 1.06577 → −0.00177) is still informative as an A/B within your setup, but the absolute val_bpb won't be directly comparable to leaderboard numbers measured on the full 47.85 M-token val set. If you can rerun on the standard val partition (the same one used for #1855's reported 1.06108), the AsymLogit delta should carry over and be easier to evaluate against the chain.

[jorge-asenjo]

@codemath3000 thanks again for catching the val truncation issue. I've now corrected and re-measured.

Root cause: the fineweb_val_000000.bin shard in our local network volume (RunPod NV) was truncated to 19 MB / 9,662,464 tokens, vs the canonical 95.7 MB / 47,851,520 tokens on Hugging Face (romeerp/parameter-golf-caseops-v1). Re-pulling directly from HF restored the full partition.

Updated submission: I've pushed a new commit (0fc7f2d) renaming the records folder and updating submission.json, README.md, and the 3 train logs:

Seed	val_bpb (full val)	val_tokens	train	eval	artifact
42	1.06030	47,851,520	599.6s	532s	15,944,044 B
0	1.05970	47,851,520	599.6s	419s	15,941,061 B
1234	1.05912	47,851,520	599.5s	457s	15,951,087 B
mean	1.05971	—	—	—	—
std	0.000478

vs PR #1855 (1.06108): −0.00137 BPB, Welch t≈4.96 (p < 0.05).

Stack also expanded: while re-running on full val I added AWQ-lite mixed-precision GPTQ (top-1 64-col group at int8) on top of AsymLogit. This converges on the same recipe @romeerp shipped in PR #1908; we landed on it independently and it stacks orthogonally with AsymLogit on full val. Credit added in the README.

submission.json carries both v1 (1.06577 truncated) and v2 (1.05971 full) revisions for transparency.

Apologies for the v1 noise. Appreciate the careful audit.