Record: 5-expert Hedge Mixer + TTT (3-seed mean val_bpb=1.0745)#687

Closed

RoyiRa wants to merge 188 commits intoopenai:mainfrom

Commits on Mar 21, 2026

docs: log mamba 80-min experiment (val_bpb=1.2728, over size limit)

RoyiRa
and
claude
committed
docs: log mamba v2 80-min result (val_bpb=1.2586, 16.0MB)

RoyiRa
and
claude
committed
docs: log mamba v3 — extreme warmdown (val_bpb=1.2565, 13.0MB)

RoyiRa
and
claude
committed
docs: log mamba v4 — 12 layers, val_bpb=1.2519 (BEST result)

RoyiRa
and
claude
committed
docs: log mamba v5 13L (1.2529) — 12L remains optimal

RoyiRa
and
claude
committed
feat(transformer): add tuned transformer baseline for experiments

RoyiRa
and
claude
committed
docs: transformer v1 beats baseline (val_bpb=1.1910, 17MB over limit)

RoyiRa
and
claude
committed
feat(transformer): add sliding window evaluation

RoyiRa
and
claude
committed
docs: transformer v2 sliding window eval (val_bpb=1.1700)

RoyiRa
and
claude
committed
docs: transformer v3 9L (val_bpb=1.1778, 15.4MB VALID)

RoyiRa
and
claude
committed
feat(transformer): add Muon weight decay (WD=0.04)

RoyiRa
and
claude
committed
docs: transformer v4 BEST (val_bpb=1.1632, 14.3MB valid)

RoyiRa
and
claude
committed
feat(transformer): add grad_clip=0.3 default

RoyiRa
and
claude
committed
docs: grad_clip=0.3 gives -0.002 bpb, warmdown=3000 > 20000

RoyiRa
and
claude
committed
fix(transformer): switch warmdown default to 3000 (from 20000)

RoyiRa
and
claude
committed
feat(transformer): add depth recurrence (num_loops parameter)

RoyiRa
and
claude
committed
docs: transformer v6 (WD=3000, clip=0.3) slightly worse than v4

RoyiRa
and
claude
committed
revert(transformer): restore warmdown=20000 (v4 config was better)

RoyiRa
and
claude
committed
docs: depth recurrence (5L×2) fails badly — 0.21 bpb worse

RoyiRa
and
claude
committed
feat(transformer): add int6 mixed quantization + zstd-22 compression

RoyiRa
and
claude
committed
fix(transformer): use simple loop when num_loops=1 for torch.compile

RoyiRa
and
claude
committed
docs: int6+zstd works but needs 80-min warmdown for quality

RoyiRa
and
claude
committed
refactor(transformer): remove depth recurrence, keep int6+zstd only

RoyiRa
and
claude
committed
docs: v9b int6+zstd saves 4.8MB but 0.034 quant damage (1.2017)

RoyiRa
and
claude
committed
feat(transformer): add late QAT + 11L + 3x MLP

RoyiRa
and
claude
committed
feat(transformer): add BigramHash + SmearGate (v11)

RoyiRa
and
claude
committed

Commits on Mar 22, 2026

Commits on Mar 23, 2026

Commits on Mar 24, 2026

Commits on Mar 25, 2026