Record: SP8192 + Polar Express NS + MIN_LR + LQER Asym Rank-4 — val_bpb 1.06766 (3-seed mean)

[AjAnubolu]

Record: PR #1790 + Polar Express NS + MIN_LR + LQER Asym Rank-4 — val_bpb 1.06766 (3-seed mean)

Summary

val_bpb = 1.06766 (3-seed mean, std 0.00076) | val_loss = 2.34122 nats/token | ~15.95 MB | 8×H100 SXM, 600s train / 600s eval

Builds directly on PR #1790's verbatim non-CaseOps stack and layers in three orthogonal, previously validated techniques from open PRs:

1. Polar Express Newton-Schulz coefficients (PR Record: SP4096 + Polar Express + MuonEq-R + Depth Recurrence — 1.0923 BPB (3-seed) #1344) — 5 per-iteration minimax-tuned tuples replace Muon's fixed (3.4445, −4.775, 2.0315) × 5, same MUON_BACKEND_STEPS=5.
2. MIN_LR=0.10 warmdown floor (PR Record: PR #1736 + Polar Express NS + MIN_LR + Sparse Attn Gate + Fused CE + PR #1767 TTT — val_bpb 1.06335 #1787) — floors LR at 10% of max so the warmdown tail delivers useful gradient updates instead of frozen no-ops.
3. LQER asymmetric rank-4 quantization correction (PR Record: PR #1787 base + Smear Gate + LQER Asym — val_bpb 1.06157 #1797) — SVD on the top-K=3 highest-error GPTQ residuals, packed as int4 with per-group asymmetric scales.

−0.00225 BPP vs PR #1790 (1.06991 → 1.06766) on the same SP8192 base, all training-time legal under Issue #1017 conditions. No CaseOps, no casefold, no preprocessing — denominator is original UTF-8 bytes throughout. Improvement over current merged SOTA (PR #1493 = 1.0810) is −0.0133 BPP at p ≪ 0.001.

3-Seed Results (8×H100 80GB SXM, phased TTT, 10-min train / 10-min eval budgets)

Seed	Steps	Pre-quant post-EMA	Quantized	Post-TTT	Artifact (bytes)	train_time	eval_time
1337	4954	1.06842	1.07813	1.06699	15,953,831	596.15s	456.6s
42	4954	1.06903	1.07856	1.06751	15,950,901	596.12s	455.2s
2025	4953	1.06994	1.07955	1.06849	15,948,627	596.13s	394.4s
Mean	4954	1.06913	1.07875	1.06766	15,951,120	596.13s	435.4s
Std		0.00076	0.00072	0.00076	2,634	0.02s	35.5s

All three seeds clear size, train-time, and eval-time budgets. 3-seed std (0.00076) is well inside the 0.005 significance floor.

Head-to-head vs PR #1790 (matched seed)

Seed	This PR	PR #1790	Δ (mBPP)
1337	1.06699	1.06986	−2.87

Single-seed comparison on seed 1337 (the only seed PR #1790 reports individually that overlaps with our set). The improvement direction is consistent at −2.25 mBPP on the 3-seed means.

Key Techniques

1. Polar Express Newton-Schulz coefficients (from PR #1344)

Muon's zeropower_via_newtonschulz5 orthogonalizes a gradient matrix via 5 iterations of a Newton-Schulz polynomial. Stock Muon uses a single fixed coefficient tuple (a, b, c) = (3.4445, −4.775, 2.0315) for all 5 iterations. Polar Express replaces this with 5 distinct per-iteration tuples, each minimax-tuned for the residual error distribution at that step. Same compute budget, sharper orthogonal approximation.

# Stock Muon
COEFS = [(3.4445, -4.775, 2.0315)] * 5

# Polar Express (PR #1344)
COEFS = [
    (a1, b1, c1),  # tuned for far-from-orthogonal input
    (a2, b2, c2),  # tuned for residual after step 1
    ...
    (a5, b5, c5),  # final polish, near-orthogonal input
]

Env var: POLAR_EXPRESS_NS=1.

2. MIN_LR=0.10 warmdown floor (from PR #1787)

PR #1790's LR schedule decays linearly to 0 during the warmdown phase. The final ~25% of training under warmdown_frac=0.75 runs at near-zero LR, delivering no useful gradient updates. Flooring lr_mul at 0.10 keeps the tail of training productive.

# PR #1790 (decay to zero)
lr_mul(step) = max((1 - frac) / warmdown_frac, 0.0)

# This PR (10% floor)
lr_mul(step) = max((1 - frac) / warmdown_frac, 0.10)

Env var: MIN_LR=0.10.

3. LQER asymmetric rank-4 quantization correction (from PR #1797)

After GPTQ quantizes a weight W, the residual E = W − dequant(quantize(W)) is non-zero. LQER computes a rank-4 SVD of E for the top-K=3 highest-error layers and packs the factors as asymmetric int4 with per-group (g=64) scales.

U, S, Vh = torch.linalg.svd(E, full_matrices=False)
A = U[:, :4] * S[:4]                # (out_dim × 4)
B = Vh[:4, :]                        # (4 × in_dim)
qA, sA = symmetric_int4_per_row(A)
qB, sB = asymmetric_int4_per_group(B, g=64)
# At inference: W_used = dequant(W_q) + dequant(qA) @ dequant(qB)

Selection by ||E||_F; in practice top-3 captures the bulk of the recoverable error mass and going wider is neutral (tested at top-12 in single-seed ablation).

Env vars: LQER_ENABLED=1 LQER_RANK=4 LQER_TOP_K=3 LQER_GROUP_SIZE=64.

Hyperparameters

SEED=<1337|42|2025> \
QK_GAIN_INIT=5.25 \
SMEAR_GATE=1 \
GATE_ATTN_OUT=1 \
GATE_ATTN_WIDTH=24 \
GPTQ_RESERVE_SECONDS=4 \
GPTQ_CALIBRATION_BATCHES=16 \
POLAR_EXPRESS_NS=1 \
MIN_LR=0.10 \
LQER_ENABLED=1 \
LQER_RANK=4 \
LQER_TOP_K=3 \
LQER_GROUP_SIZE=64 \
torchrun --standalone --nproc_per_node=8 train_gpt.py

All other hyperparameters inherited from PR #1790 base unchanged.

Rule Compliance (Issue #1017 Track B)

1. Strict causal dependence: LoRA state built from prefix tokens only. Phased global SGD at boundaries operates on already-scored prefix docs only (inherited from PR Record: SP8192 + SmearGate + AttnOutGate(w24) + LoRA-TTT Improvements + Phased TTT — val_bpb 1.06991 (3-seed mean) #1790).
2. Full normalized distribution: standard softmax over SP8192 vocab. No n-gram cache, no logit biasing.
3. Score before update: per-chunk forward pass and loss accumulation BEFORE any LoRA gradient step. Last chunk of each phase explicitly skipped.
4. Single left-to-right pass: each token scored exactly once, no rescoring.

• Artifact ≤ 16,000,000 bytes: ✅ all seeds 15.95 MB
• Training ≤ 600s: ✅ all seeds 596.1s (wallclock-capped, identical to PR Record: SP8192 + SmearGate + AttnOutGate(w24) + LoRA-TTT Improvements + Phased TTT — val_bpb 1.06991 (3-seed mean) #1790)
• Eval ≤ 600s: ✅ all seeds 394–457s
• No val data during training: ✅ training uses only fineweb_train_*.bin shards
• 3-seed validation: ✅
• BPB on original UTF-8 bytes: ✅ no preprocessing, no sidecar required
• Reproducible: ✅ deterministic given seed and the env vars above

Attribution

• @miaoyuxun — PR Record: SP8192 + SmearGate + AttnOutGate(w24) + LoRA-TTT Improvements + Phased TTT — val_bpb 1.06991 (3-seed mean) #1790 base stack (SmearGate + AttnOutGate(w24) + LoRA-TTT improvements + Phased TTT)
• @bigbag — PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 prior merged SOTA (SP8192 base)
• @MarioPaerle — PR RECORD: SmearGate + Attention Output Gate + Legal TTT | val_bpb=1.07139 #1667 SmearGate + AttnOutGate (inherited via PR Record: SP8192 + SmearGate + AttnOutGate(w24) + LoRA-TTT Improvements + Phased TTT — val_bpb 1.06991 (3-seed mean) #1790)
• @renqianluo — PR Record: Alpha=144 LoRA + Warm-start A + WD 1.0 — val_bpb 1.07209 (3-seed mean) #1767 LoRA TTT improvements (inherited via PR Record: SP8192 + SmearGate + AttnOutGate(w24) + LoRA-TTT Improvements + Phased TTT — val_bpb 1.06991 (3-seed mean) #1790)
• @jorge-asenjo — PR Add SP8192 Multi-Phase Global SGD + Phased TTT (1.07219 bpb) #1700 phased TTT (inherited via PR Record: SP8192 + SmearGate + AttnOutGate(w24) + LoRA-TTT Improvements + Phased TTT — val_bpb 1.06991 (3-seed mean) #1790)
• Polar Express Newton-Schulz coefficients — PR Record: SP4096 + Polar Express + MuonEq-R + Depth Recurrence — 1.0923 BPB (3-seed) #1344
• MIN_LR warmdown floor — PR Record: PR #1736 + Polar Express NS + MIN_LR + Sparse Attn Gate + Fused CE + PR #1767 TTT — val_bpb 1.06335 #1787
• LQER asymmetric rank-4 — PR Record: PR #1787 base + Smear Gate + LQER Asym — val_bpb 1.06157 #1797

Test plan

• Organizer reviews submission folder contents (train_gpt.py, 3 seed logs, submission.json, README.md).
• Organizer reproduces at least one seed: SEED=1337 ... torchrun --standalone --nproc_per_node=8 train_gpt.py (full env above).
• Reproduced quantized_ttt_phased val_bpb matches the logged 1.06699 (±0.00076) within seed noise.
• Artifact size, train_time, total_eval_time within budgets on re-run.

[andrewbaggio1]

Potential SmearGate boundary leak in #1874 / inherited #1790 path

I noticed this PR appears to inherit the pre-BOS-fix SmearGate pattern from the #1790 stack. In the PR diff, both forward_logits and forward have the same shape:

g = sl * torch.sigmoid(self.smear_gate(x[:, 1:, :self.smear_width]))
x = torch.cat([x[:, :1], x[:, 1:] + g * x[:, :-1]], dim=1)

If production train/eval ever passes packed multi-document rows, then for a packed stream

[..., a, BOS, b1, b2, ...]

the BOS embedding for the new document becomes

E[BOS] + gate(E[BOS]) * E[a]

and the logit at that BOS position scores b1, so document B can depend on document A's final token before attention/cu_seqlens can reset anything.

Could you confirm whether #1874's production train/eval path ever presents packed multi-doc rows to this SmearGate code? If yes, this needs the same BOS mask used in the BOS-fix submissions:

bos = input_ids[:, 1:].eq(BOS_ID)
prev = x[:, :-1].masked_fill(bos[..., None], 0)
x = torch.cat([x[:, :1], x[:, 1:] + g * prev], dim=1)

This is independent of CaseOps; it is a packed-document boundary issue.