Record: L-BFGS SLOT — val_bpb 0.5793 (3-seed mean)#675

Closed

ChideraIbe123 wants to merge 108 commits intoopenai:mainfrom

ChideraIbe123:main

Commits on Mar 19, 2026

Implement recursive transformer with per-loop LoRA deltas

authored and
Chidera Ibe
committed
Fix GQA compatibility with PyTorch 2.4 (no enable_gqa arg)

authored and
Chidera Ibe
committed
Fix convergence: smaller model, fewer loops, non-zero LoRA init

authored and
Chidera Ibe
committed
3 shared blocks × 3 loops at dim 768 (9 effective layers)

authored and
Chidera Ibe
committed
Fix instability: zero LoRA B init, lower matrix_lr for shared blocks

authored and
Chidera Ibe
committed
Restore native enable_gqa (PyTorch upgraded on RunPod)

authored and
Chidera Ibe
committed
Pivot to baseline + proven improvements

authored and
Chidera Ibe
committed
Fix LAWA: only collect checkpoints from last half of warmdown

authored and
Chidera Ibe
committed

Commits on Mar 21, 2026

Add sliding window eval + TTT at eval time

authored and
Chidera Ibe
committed
Increase eval stride 64->512 (64 too slow on 1xH100)

authored and
Chidera Ibe
committed
Disable slow evals by default, focus on QAT next

authored and
Chidera Ibe
committed
Add entropy-weighted training loss (novel technique)

authored and
Chidera Ibe
committed
Revert entropy loss, add QAT (fake int8 quantize in CastedLinear)

authored and
Chidera Ibe
committed
Add ramping weight decay (0.02→0.08 during warmdown)

authored and
Chidera Ibe
committed
Disable QAT, keep ramping WD only

authored and
Chidera Ibe
committed
Add 10th layer (3.5MB headroom from WD compression)

authored and
Chidera Ibe
committed
Bump to 11 layers (2.3MB headroom remaining)

authored and
Chidera Ibe
committed
Add 3x MLP expansion (from SOTA PR #287)

authored and
Chidera Ibe
committed

Commits on Mar 25, 2026

Drop to 10 layers (11L+3xMLP=18.3MB, over budget)

authored and
Chidera Ibe
committed
Drop to 9L+3xMLP (10L+3xMLP=16.77MB, over budget)

authored and
Chidera Ibe
committed
Revert to best config: 10L + 2x MLP (1.2405 BPB)

authored and
Chidera Ibe
committed
Add LeakyReLU², lzma compression, 5-gram eval cache

authored and
Chidera Ibe
committed
Add Differential Attention (ICLR 2025, arXiv:2410.05258)

authored and
Chidera Ibe
committed
Use Flash Attention for Differential Attention (2x speedup)

authored and
Chidera Ibe
committed
Fix SDPA dim mismatch: split V into halves too, concat after

authored and
Chidera Ibe
committed
Revert to Exp 16 best config (1.2302 BPB)

authored and
Chidera Ibe
committed
Add Value Residual Learning (VRL, ACL 2025, arXiv:2410.17897)

authored and
Chidera Ibe
committed
Remove 5-gram eval cache (too slow, takes 30+ min on 1xH100)

authored and
Chidera Ibe
committed
Revert to Exp 16 best config (1.2302 BPB) — remove VRL

authored and
Chidera Ibe
committed
Remove 5-gram cache again (came back with revert)

authored and
Chidera Ibe
committed
Non-record: LeakyReLU² + LAWA + Ramping WD + Val Training (val_bpb=1.2302, 1xH100)

authored and
Chidera Ibe
committed
Add Harmonic Loss blend (arXiv:2502.01628, Feb 2025)

authored and
Chidera Ibe
committed
Fix: only use harmonic blend during training, eval uses pure CE

authored and
Chidera Ibe
committed
Revert to Exp 16 best config (1.2302 BPB), remove 5-gram cache

authored and
Chidera Ibe
committed
Add LayerDrop via residual dropout (arXiv:1909.11556, DropPEFT 2025)

authored and
Chidera Ibe
committed

Commits on Mar 26, 2026

Revert to Exp 16 best config, remove 5-gram cache permanently

authored and
Chidera Ibe
committed
Replace transformer with Mamba SSM (arXiv:2312.00752, Mamba-3: 2603.15569)

authored and
Chidera Ibe
committed
Fix Mamba OOM: use sequential scan instead of parallel cumsum

authored and
Chidera Ibe
committed
Revert to Exp 16 best config — Mamba too slow without CUDA kernels

authored and
Chidera Ibe
committed
Add Gated Attention (NeurIPS 2025 Oral, arXiv:2505.06708)

authored and
Chidera Ibe
committed
Add Progressive Residual Warmup / ProRes (arXiv:2603.05369, March 2026)

authored and
Chidera Ibe
committed
Fix ProRes: use registered buffer instead of global (torch.compile safe)

authored and
Chidera Ibe
committed
Tune ProRes: faster warmup (30+30*l instead of 100+80*l)

authored and
Chidera Ibe
committed
Fix QAT: match actual quantizer (per-row clip) + enable at 50%

authored and
Chidera Ibe
committed
Revert to Exp 26 best (1.2287 BPB) — QAT still doesn't help

authored and
Chidera Ibe
committed
Update submission with Exp 26 ProRes best config (1.2287 BPB)

authored and
Chidera Ibe
committed

Commits on Mar 28, 2026

Tune hyperparams from SOTA submissions (zero overhead)

Chidera Ibe
and
claude
committed
Revert warmdown to 1200 (2000 covers entire run on 1xH100)

Chidera Ibe
and
claude
committed
Revert hyperparams to defaults (SOTA settings need 7K+ steps)

Chidera Ibe
and
claude
committed
Add Deep Delta Learning (arXiv:2601.00417, Jan 2026)

Chidera Ibe
and
claude
committed
Fix DDL: per-token transform instead of mean-pooled (gradient shape fix)

Chidera Ibe
and
claude
committed

Commits on Apr 3, 2026

SOTA base (#1263, 0.9354) + ProRes (arXiv:2603.05369)

authored and
Chidera Ibe
committed
Tune ProRes for 8xH100 (100+100*l) + need zstd install

authored and
Chidera Ibe
committed
Remove ProRes, add DyT (Dynamic Tanh, arXiv:2503.10622, March 2026)

authored and
Chidera Ibe
committed
Compact DyT to stay under 1500 line limit

authored and
Chidera Ibe
committed
Revert DyT to RMSNorm + SGD momentum SLOT (novel eval improvement)

authored and
Chidera Ibe
committed
Make SLOT optimizer configurable via SLOT_OPTIMIZER env var

authored and
Chidera Ibe
committed
New SOTA base (#1313, 0.8637) + Hypergradient SLOT (arXiv:2502.11229)

authored and
Chidera Ibe
committed
Record: Hypergradient SLOT-24 — val_bpb 0.7625 (3-seed mean)

authored and
Chidera Ibe
committed
Add 3-seed training logs

Chidera Ibe
authored and
Chidera Ibe
committed

Commits on Apr 4, 2026

Add LoRA-SLOT: context-dependent delta via low-rank matrices (novel)

authored and
Chidera Ibe
committed

Commits on Apr 6, 2026

Replace LoRA-SLOT with Entropy-Gated SLOT (novel)

Chidera Ibe
and
claude
committed
Warm-Restart Hypergradient SLOT: 48 steps with optimizer reset

authored and
Chidera Ibe
committed
L-BFGS SLOT: second-order optimizer for per-sample delta (novel)

authored and
Chidera Ibe
committed
L-BFGS 8 steps + skip diagnostic evals (fit in 600s budget)

authored and
Chidera Ibe
committed
Record: L-BFGS SLOT-8 — val_bpb 0.5793 (3-seed mean)

Chidera Ibe
authored and
Chidera Ibe
committed
Add n-gram cache on top of L-BFGS SLOT (stack orthogonal techniques)

authored and
Chidera Ibe
committed
Revert to pure L-BFGS SLOT (0.5793) — n-gram too slow (1411s vs 600s limit)

Chidera Ibe
authored and
Chidera Ibe
committed
Warm-start L-BFGS SLOT: init from previous batch + 10 steps

authored and
Chidera Ibe
committed

Commits on Apr 7, 2026

Add pre-quant TTT + L-BFGS SLOT combination

Chidera Ibe
authored and
Chidera Ibe
committed
Fix pre-quant TTT: batch=8, freeze 2 blocks, cosine LR, 3 epochs

Chidera Ibe
authored and
Chidera Ibe
committed
Revert to pure L-BFGS SLOT (0.5793) — pre-quant TTT hurt (0.5835)

Chidera Ibe
authored and
Chidera Ibe
committed
Scored-positions-only L-BFGS: 21x less compute per step, 20 steps

Chidera Ibe
authored and
Chidera Ibe
committed
Revert to clean 8-step L-BFGS SLOT (0.5793)

Chidera Ibe
authored and
Chidera Ibe
committed
Add warmdown label smoothing (ICML 2025, arXiv:2508.00264)

Chidera Ibe
authored and
Chidera Ibe
committed
Revert label smoothing — disrupted training (0.6136 vs 0.5793)

Chidera Ibe
authored and
Chidera Ibe
committed
Scored-positions L-BFGS + order-12 n-gram hash mixer

Chidera Ibe
authored and
Chidera Ibe
committed
L-BFGS SLOT + vectorized n-gram mixer (from PR #1430)

Chidera Ibe
authored and
Chidera Ibe
committed
Reduce to 6 L-BFGS steps (8+ngram=718s, over budget)

Chidera Ibe
authored and
Chidera Ibe
committed

Commits on Apr 8, 2026

N-gram order 12→18, buckets 4M→8M (longer context, fewer collisions)

Chidera Ibe
authored and
Chidera Ibe
committed
Hedge multi-expert mixing + fix K indexing

Chidera Ibe
authored and
Chidera Ibe
committed
Revert to 0.2968 config (6-step L-BFGS + order-12 backoff ngram)

Chidera Ibe
authored and
Chidera Ibe
committed
Hedge neural/ngram mixing on order-12 backoff

Chidera Ibe
authored and
Chidera Ibe
committed
Revert to 0.2968 backoff (Hedge hurt: 0.4297)

Chidera Ibe
authored and
Chidera Ibe
committed
Log-space Hedge mixing (PR #688 style)

Chidera Ibe
authored and
Chidera Ibe
committed
Quality-weighted alpha: use n-gram order + count for better mixing

Chidera Ibe
authored and
Chidera Ibe
committed
Confidence-boosted alpha: trust n-gram more when it's confident

Chidera Ibe
authored and
Chidera Ibe
committed
Increase confidence boost 0.15->0.25

Chidera Ibe
authored and
Chidera Ibe
committed
Boost 0.25->0.30

Chidera Ibe
authored and
Chidera Ibe
committed
Boost 0.30->0.40

Chidera Ibe
authored and
Chidera Ibe
committed
Boost 0.40->0.50

Chidera Ibe
authored and
Chidera Ibe
committed
Boost 0.50->0.60

Chidera Ibe
authored and
Chidera Ibe
committed
Boost 0.60->0.70

Chidera Ibe
authored and
Chidera Ibe
committed
Boost 0.70->0.80

Chidera Ibe
authored and
Chidera Ibe
committed
Fix: target-independent alpha (order + context count features)

Chidera Ibe
authored and
Chidera Ibe
committed
Tune: order 0.20->0.35, count 0.10->0.20

Chidera Ibe
authored and
Chidera Ibe
committed
Tune alpha: order=0.55, count=0.30

authored and
Chidera Ibe
committed

Commits on Apr 9, 2026

Revert alpha to best: order=0.35, count=0.20 (0.2280 BPB)

authored and
Chidera Ibe
committed
Implement In-Place TTT (arXiv:2604.06169)

authored and
Chidera Ibe
committed
Fix In-Place TTT bugs found via paper verification

authored and
Chidera Ibe
committed
Tune TTT: stronger init + higher eta + looser clip

authored and
Chidera Ibe
committed
Fix torch.compile: replace data-dependent if with torch.clamp

authored and
Chidera Ibe
committed
Fix TTT eval dtype (bf16 conv vs float x0) + reduce to 1 layer

authored and
Chidera Ibe
committed
Disable TTT: hurts BPB (0.2289 vs 0.2282) and blows eval budget (640s)

authored and
Chidera Ibe
committed
SLOT warm-starting: init delta from previous window's optimum

Chidera Ibe
and
claude
committed
Fix n-gram count deduplication: only update with new tokens

authored and
Chidera Ibe
committed
Drop n-gram mixer: normalization issue affects all hash-based n-gram PRs

authored and
Chidera Ibe
committed
Revert to clean L-BFGS SLOT (PR #675 version, 0.5793 BPB)

authored and
Chidera Ibe
committed