Record: SP8192 + CaseOps + GatedAttn + QuantGate + Loop45 + PhasedTTT — val_bpb 1.06549

[dexhunter]

Summary

• val_bpb 1.06549 (3-seed mean, std 0.00070), val_loss 2.33168 nats/token.
• Builds on PR Record: Varlen attention + fused MLP + doc-independent TTT (1.07336) #1530 SP8192 stack. Adopts CaseOps (lossless_caps_caseops_v1) bijective case preprocessing from PR Record: CaseOps Tokenizer + Tapered WD - val_bpb 1.0678 (3-seed mean) #1729 with a per-token byte sidecar so BPB is scored on original pre-transform UTF-8 bytes.
• Adds a learned attention out-gate (init_std=0.005) + quant-gate scaling that recovers the overhead introduced by the new control tokens, keeping every seed under the 16 MB decimal cap.

3-seed results (8×H100 80GB SXM, 10-min train / 10-min eval budgets)

Seed	Steps	Pre-TTT BPB	Post-TTT BPB	Artifact (bytes)	train_time	eval_time
42	4854	1.07847	1.06610	15,978,834	596.18s	396.9s
0	4843	1.07719	1.06473	15,971,476	596.17s	399.3s
1234	4847	1.07811	1.06563	15,975,050	596.08s	395.5s
Mean	4848	1.07792	1.06549	15,975,120	596.14s	397.23s
Std		0.00066	0.00070	3,698	0.06s	1.9s

All three seeds clear size, train-time, and eval-time budgets with substantial headroom. 3-seed std is 0.00070 BPB — well inside the 0.005 significance floor.

Key innovation — CaseOps tokenizer + byte sidecar

CaseOps is a bijective, character-level text transform that removes English capitalization from the body of the text and records it as four operator tokens (TITLE, ALLCAPS, CAPNEXT, ESC) that become SentencePiece user_defined_symbols. Because the transform is fully invertible (decode(encode(s)) == s), no information is lost and BPE merges allocate vocabulary around content instead of around case variants. Ships with a per-token byte sidecar (fineweb_val_bytes_*.bin, uint16 parallel to val shards) so BPB is computed on ORIGINAL pre-transform UTF-8 bytes, not on the transformed representation — the score is on the same FineWeb text, just with a different tokenization front end.

Rule compliance

• Artifact ≤ 16,000,000 bytes DECIMAL (README FAQ + Issue A Field Guide to Valid Submissions #1017 §II.1): ✅ all seeds ≤ 15,978,834.
• train_time ≤ 600s (README line 6): ✅ all seeds 596.1s.
• total_eval_time ≤ 600s (README FAQ, separate budget): ✅ all seeds 395.5–399.3s.
• Score-first TTT (Issue A Field Guide to Valid Submissions #1017 Condition 3): ✅ phased TTT snapshots the pre-update score on each chunk before the LoRA adapter step; per-doc LoRA reset between documents.
• BPB on original bytes (Issue A Field Guide to Valid Submissions #1017 §V): ✅ per-token byte sidecar encodes canonical UTF-8 byte count of each val position.
• No val data in training: ✅ training uses only fineweb_train_*.bin shards.
• Reproducible: prepare_caseops_data.py is deterministic given the input FineWeb doc stream.

Test plan

• Organizer reviews submission folder contents (train_gpt.py, prepare_caseops_data.py, tokenizer .model, 3 seed logs, submission.json, README.md, lossless_caps.py).
• Organizer runs prepare_caseops_data.py to generate CaseOps shards + val byte sidecar.
• Organizer reproduces at least one seed: SEED=42 CASEOPS_ENABLED=1 GATED_ATTN_QUANT_GATE=1 ... torchrun --standalone --nproc_per_node=8 train_gpt.py (full env in README).
• Reproduced quantized_ttt_phased val_bpb matches the logged 1.06610 (±0.0007) within seed noise.
• Artifact size, train_time, total_eval_time all within budgets on re-run.

Lineage

• @samacqua — PR Record: Varlen attention + fused MLP + doc-independent TTT (1.07336) #1530 base stack.
• @romeerp — PR Record: CaseOps Tokenizer + Tapered WD - val_bpb 1.0678 (3-seed mean) #1729 CaseOps concept + byte sidecar accounting.
• @MarioPaerle — PR RECORD: SmearGate + Attention Output Gate + Legal TTT | val_bpb=1.07139 #1667 attention gate pattern.
• @bigbag — PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 prior merged SOTA.

[codemath3000]

Running prepare_caseops_data.py as published, then running train_gpt.py with PHASED_TTT_ENABLED=1 reproducibly raises ZeroDivisionError: float division by zero at train_gpt.py:2303 in _loss_bpb_from_sums — byte_sum.item() is 0 because _find_docs (line 2209) returns an empty list. The prep script never inserts BOS markers, and the tokenizer reserves IDs 0–7 (<pad>, <s>, </s>, <unk>, and the four CaseOps operators), so sp.encode can never naturally output id 1. The training loop has a fallback at _init_shard line 408-409 (if self.bos_idx.size == 0: self.bos_idx = np.array([0], ...)) so training completes, but the phased TTT eval path has no analogous fallback. Am I missing a prep step, or should prepare_caseops_data.py be prepending bos_id=1 to each doc (matching download_hf_docs_and_tokenize.py:364-366)?

[dexhunter]

@codemath3000 thanks for the reproducer — confirmed and patched. This is a prep-script bug only; training and the submitted metric are unaffected.

Root cause. Line 157 of prepare_caseops_data.py calls sp.encode(transformed) and appends directly to the shard buffer. The SP tokenizer reserves IDs 0–7 (<pad>, <s>, </s>, <unk>, TITLE, ALLCAPS, CAPNEXT, ESC), so sp.encode cannot emit BOS (ID 1) naturally. _find_docs at train_gpt.py:2209 then returns [] and _loss_bpb_from_sums at line 2303 divides by zero. Training survives via the _init_shard:408–409 fallback; phased TTT eval has no analogous fallback.

Scope. The submitted 1.06549 is on valid data — our seed runs used shards produced by a different internal prep path that already prepends BOS. val_bpb reduces to loss_sum / ln(2) / byte_sum (token counts cancel at line 2303) and byte_sum is unchanged with BOS prepended (BOS contributes 0 original bytes). The bug broke reproduction, not the number.

Fix. Prepend BOS_ID = 1 to each doc's tokens and append 0 to the byte-count sidecar for the BOS position:

# near module top, with other constants
BOS_ID = 1

# inside the per-doc loop
for text in _iter_docs(args.docs):
    transformed = encode_lossless_caps_v2(text)
    token_ids = [BOS_ID] + sp.encode(transformed, out_type=int)
    if n_docs < args.val_docs:
        byte_counts = _token_original_byte_counts(sp, text, transformed)
        val_buf_tokens.extend(token_ids)
        val_buf_bytes.append(0)  # BOS = 0 original bytes
        val_buf_bytes.extend(int(b) for b in byte_counts)
    else:
        train_buf.extend(token_ids)

Matches the canonical pattern in data/download_hf_docs_and_tokenize.py:364–366.

Pushed in commit d7263a3 on this branch (and fe7c309 on PR #1769, which ships the same prep script). README now includes a bos_count > 0 sanity check for the first val shard.