Record: K_KVShare_Wider full-recipe FLA — val_bpb 1.04090 (3-seed mean)

[resouer]

Summary

3-seed mean val_bpb: 1.04089763 (std 0.00106003) | 3.16476760 nats | 8xH100 SXM, 600s

Seed	Quantized BPB	val_loss (nats)	Artifact
1337	1.03967403	3.16104735	15,762,406
42	1.04153708	3.16671180	15,870,797
2025	1.04148177	3.16654364	15,648,800
Mean	1.04089763	3.16476760	15,760,668

Mechanism

This candidate packages the stronger K_KVShare_Wider point on a fuller upstream-style FLA / GatedDeltaNet recipe.

Main idea:

• FLA / GatedDeltaNet family
• K_KVShare_Wider: KV sharing (kv_sharing_stride=2) used to buy width rather than depth
• fuller upstream-style recipe
• EMA + SWA + late QAT + int6 artifact path

Nearest prior family reference: PR #1370.

Compliance notes

What this candidate does not use:

• no TTT
• no SLOT
• no n-gram overlay
• no SWA/XSA final scoring path (K_KVShare_Wider has num_swa_layers = 0)

Hardening already applied:

• all 3 seeds are on one script revision
• train_gpt.py does not perform runtime dependency downloads
• dependencies are declared in requirements.txt and expected to be installed before evaluation

Packaging note

This PR keeps the faithful multi-file records surface rather than the tidier single-file experiments.

A direct packaged smoke on this exact multi-file surface (seed 1337) completed at 1.03971272 BPB, versus the measured seed1337 result of 1.03967403, a delta of only +0.00003869 BPB. Peak memory remained 41127 MiB.

Packaged-folder verification

Exact draft records-folder audit:

• packaged folder + largest artifact = 15,991,282 bytes
• remaining headroom = 8,718 bytes

The packaged code in the records folder passes py_compile from inside the folder.

Reproduction

pip install --no-deps -r requirements.txt
SEED=$SEED ARCH_MODE=K MAX_WALLCLOCK_SECONDS=600 VAL_LOSS_EVERY=0 EVAL_COMPILE_ENABLED=0 \
  torchrun --standalone --nproc_per_node=8 train_gpt.py

[bigbag]

Hi @resouer — I think this PR hits the same build_sentencepiece_luts byte-counting bug that closed #1545 / #1576 / #1632 / #1672 / #1681. Flagging for visibility.

Where: train_gdn_7k.py L192–205 sets base_bytes[i] = len(piece[1:].encode("utf-8")) + 1 for ▁-prefixed pieces — so the leading-space byte is already baked in. Then the scorer at L361–363 adds another +1 via has_leading_space_lut[tgt] & ~is_boundary_token_lut[prev]. Every ▁-token ends up counted with one extra byte.

Reference (base train_gpt.py L180–204) strips the ▁ from piece before calling .encode("utf-8"), so base_bytes counts only the content and the scorer's +1 is the only leading-space accounting.

Empirical check on fineweb_1024_bpe.model with a 3,648-byte FineWeb-style sample (reproduces the scorer arithmetic from L361–363):

	actual text	reference LUT	this PR's LUT
byte_count	3,648	3,647	4,214 (+15.55%)

Per-token delta (PR − ref): 68.0% of tokens → 0, 31.5% → +1 (the ▁-tokens), 0.5% → +5 (looks like sp.is_byte tokens, where the reference uses base_bytes = 1 but this impl uses len("<0x..>".encode())).

Correction factor ×1.1555 → the reported 1.0409 3-seed mean becomes ≈ 1.20 under the reference scorer, which is in line with the prior GDN-Hybrid runs before rescoring.

Minimal fix, mirroring base train_gpt.py L180–204:

• strip ▁ from piece before piece.encode("utf-8")
• default is_boundary_token to True; set False only for non-control/unknown/unused tokens
• handle sp.is_byte(i) → base_bytes = 1
• add sp.is_unused(i) to the control/unknown skip

Happy to share the verification script if useful.

[resouer]

Closing this submission after local verification found a scoring bug in the SentencePiece byte accounting path. The corrected rerun on commit 3bf1ec3 lands at final_int6_roundtrip_exact val_bpb:1.22285488, so the reported 1.04089763 result does not hold under the base scorer semantics.