Record: CaseOps Tokenizer + Tapered WD - val_bpb 1.0678 (3-seed mean)

[romeerp]

Summary

• val_bpb: 1.06780 (3-seed mean, std 0.00037) | ~15.94 MB | 8xH100 80GB SXM
• Builds on PR Record: VarLen Attention + Fused MLP + Multi-Phase Global SGD TTT — val_bpb 1.07193 (3-seed mean) #1626's legal multi-phase TTT stack
• Replaces standard sp8192 with a lossless CaseOps tokenizer / dataset export hosted publicly at romeerp/parameter-golf-caseops-v1
• Adds a mild late Muon WD taper: WD_TAPER_START_FRAC=0.70, WD_TAPER_FINAL_MULT=0.50
• Keeps phased TTT legal and score-first while improving pretrained, quantized, and post-TTT BPB

Results

Seed	Steps	Pre-Quant BPB	Quantized BPB	Post-TTT BPB	Artifact
0	4,921	1.07032992	1.08152131	1.06805820	15,932,307
42	4,866	1.07065549	1.08171495	1.06806595	15,935,802
1234	4,870	1.06971629	1.08036614	1.06727867	15,943,106
Mean		1.07023390	1.08120080	1.06780094	15,937,072

All 3 seeds are under the 600s train budget, under the 600s eval budget, and under the 16 MB artifact cap.

Method

This submission combines two ideas:

1. Lossless CaseOps tokenizer

   • Uses the reversible lossless_caps_caseops_v1 transform.
   • Factorizes text into a lowercase lexical stream plus a tiny capitalization side channel (TITLE, ALLCAPS, CAPNEXT, ESC).
   • Original text is reconstructed exactly by replaying those capitalization operators over the lowercase stream, so no information is discarded.
   • This reduces redundant case fragmentation in the main token stream while preserving exact recoverability.
   • Validation BPB is still charged against exact original UTF-8 bytes using exported validation byte sidecars.

2. Mild tapered weight decay

   • Keeps full Muon WD early in training when regularization/compressibility pressure matters most.
   • Tapers to half the base WD late in training, where weights are more settled and the optimization benefit appears to outweigh the regularization benefit.

The base architecture and legal phased-TTT evaluation flow come from PR #1626; this submission changes the tokenizer/data path and late WD schedule.

Why the tokenizer is still metric-correct

The exporter writes:

• fineweb_val_000000.bin
• fineweb_val_bytes_000000.bin

The trainer loads the byte sidecar directly and logs:

• val_bpb:byte_sidecar:enabled

So BPB is computed against original raw-byte counts, not the transformed token stream length.

Legality

• Attention remains causal.
• Scoring uses standard normalized cross-entropy.
• Phased TTT remains score-first in the PR Record: VarLen Attention + Fused MLP + Multi-Phase Global SGD TTT — val_bpb 1.07193 (3-seed mean) #1626 sense.
• No validation token is used for adaptation before its score is counted.
• The tokenizer change is legality-preserving because it is a fully reversible preprocessing transform applied uniformly before tokenization.
• No information is discarded: the original text can be reconstructed exactly from the lowercase stream plus the capitalization operators.
• BPB is still charged against the original raw UTF-8 bytes through the exported validation byte sidecar, not against transformed text length.

Reproducibility

Public artifacts:

• Dataset + tokenizer: romeerp/parameter-golf-caseops-v1

The record folder includes:

• train_gpt.py
• README.md
• submission.json
• requirements.txt
• cached_challenge_fineweb.py
• download_hf_docs_and_tokenize.py
• lossless_caps.py
• tokenizer_specs_export_caseops_v1_reserved_only.json
• all 3 seed logs

Run instructions

From the record directory:

cd records/track_10min_16mb/2026-04-18_PR1626_CaseOps_Taper

Prepare the public HF tokenizer + dataset:

MATCHED_FINEWEB_REPO_ID=romeerp/parameter-golf-caseops-v1 \
MATCHED_FINEWEB_REMOTE_ROOT_PREFIX=datasets \
python3 cached_challenge_fineweb.py \
  --variant sp8192_lossless_caps_caseops_v1_reserved \
  --train-shards 80

Train + quantize + phased eval for one seed:

NCCL_NET=Socket \
SEED=0 \
TOKENIZER_PATH=./tokenizers/fineweb_8192_bpe_lossless_caps_caseops_v1_reserved.model \
DATASETS_DIR=./datasets/fineweb10B_sp8192_lossless_caps_caseops_v1_reserved \
torchrun --standalone --nproc_per_node=8 train_gpt.py \
  > train_seed0.log 2>&1