Byte-accounting bug in `build_sentencepiece_luts` affects GDN-family submissions

[dexhunter]

Summary

Several open GDN/FLA-family submissions report val_bpb figures that are inconsistent with Section V of the README. Root cause is a shared build_sentencepiece_luts helper that double-credits the leading-space byte for ▁-prefixed SentencePiece tokens, inflating the byte denominator by ≈17% and proportionally deflating the reported val_bpb.

Defect

Example from PR #1712, train_gdn_7k.py:204-217:

if piece.startswith("\u2581"):
    has_space[i] = True
    base_bytes[i] = len(piece[1:].encode("utf-8")) + 1   # pre-credits +1

The eval accumulator (PR #1712, train_gdn_7k.py:373-375) then adds +1 a second time:

tb = base_bytes_lut[tgt].to(torch.float64)
tb += (has_leading_space_lut[tgt] & ~is_boundary_token_lut[prev]).to(torch.float64)   # +1 again

For every ▁-prefixed target following a non-boundary token, one byte is counted twice.

Canonical reference

Merged PR #1019 — records/track_10min_16mb/2026-03-25_ValCalib_GPTQ_XSA_BigramHash3072/train_gpt.py:266-290:

if piece.startswith("\u2581"):
    has_leading_space_np[token_id] = True
    piece = piece[1:]                                    # strip ▁ first
base_bytes_np[token_id] = len(piece.encode("utf-8"))     # no +1 in LUT

The +1 is applied exactly once, in the eval accumulator.

Quantified impact (sp8192 val stream, 40,540,160 tokens)

Scheme	Total val bytes	Bytes/token
Buggy	177,825,759	4.386
Canonical	~151–152M	~3.73
Inflation	+17.7%	—

Self-check: the ratio val_loss / val_bpb should be ≈2.58 for sp8192 (≈3.73 bytes/token × ln 2); a ratio ≈3.04 indicates the bug.

Affected submissions

PR	Reported val_bpb	Canonical estimate
#1576 (closed)	1.0167	—
#1632 (closed)	1.028	~1.20
#1698	1.00995	~1.177
#1711	1.00980	~1.177
#1712	1.01902	~1.200

All LUTs examined contain the byte-identical len(piece[1:].encode("utf-8")) + 1 pattern. After correction, none of the listed submissions surpass the merged SOTA (PR #1493 at 1.0810).

Recommended action

1. Affected authors: replace base_bytes[i] = len(piece[1:].encode("utf-8")) + 1 with base_bytes[i] = len(piece[1:].encode("utf-8")) and re-evaluate. The eval accumulator already adds the +1 via has_leading_space_lut & ~is_boundary_token_lut.
2. Evaluation pipeline: consider a semantic check that rejects any submission where the byte LUT pre-credits the leading-space byte while the eval accumulator also adds it.
3. New submissions: derive build_sentencepiece_luts from merged PR Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019 rather than inheriting from family source files.

[dexhunter]

Update — the bug is propagating through new forks faster than the original affected submissions are being fixed.

PR #1698 (@arsenis-cmd, 1.00995 claimed): confirmed same bug at train_gdn_7k.py:1446-1459 + eval mask at 1615-1616 / 1735-1736. Canonical ≈ 1.189. Also fails the 16 MB artifact cap (16.47-16.60 MB across 3 seeds).

PR #1734 (@yahya010, 1.01080 claimed): a compression-swap fork of PR #1698 that successfully fixes the artifact cap (brotli-11 swap, all 3 seeds at ~15.5 MB) but inherits train_gdn_7k.py verbatim including the buggy byte LUT:

# PR #1734 diff line ~1405
base_bytes[i] = len(piece[1:].encode("utf-8")) + 1   # pre-credits +1

# PR #1734 diff lines ~1565, ~1739 — eval adds +1 AGAIN
tb += (has_leading_space_lut[tgt] & ~is_boundary_token_lut[prev]).to(torch.float64)

Canonical ≈ 1.01080 × 1.177 = 1.190, under merged SOTA (1.0810) by ~0.109 BPB.

Updated affected-submissions table:

PR	Author	Reported val_bpb	Canonical estimate	Status
#1576 (closed)	—	1.0167	—	organizer-flagged for this bug
#1632 (closed)	@resouer	1.028	~1.20	same family
#1687	—	—	—	parent file of #1712
#1698	@arsenis-cmd	1.00995	~1.189	BPB bug + artifact >16 MB
#1711	—	1.00980	~1.177	same family
#1712	@aamodbhatt	1.01902	~1.200	same family
#1734	@yahya010	1.01080	~1.190	brotli fix retains LUT bug

The inheritance chain is the real pattern here — once one author's train_gdn_7k.py is forked, the bug carries across. A one-line semantic check in the evaluation pipeline would break it: reject submissions where the byte LUT pre-credits the leading-space byte AND the eval accumulator adds +1 for ▁-prefixed, non-boundary targets.

[HKati]

This looks like a useful example of why byte-accounting evidence should be surfaced explicitly at review time, rather than inferred later from implementation details or inherited helper code.

For submissions that introduce, modify, or inherit tokenization or byte-accounting logic, including tokenizer-derived LUTs, a small review-side receipt could include:

• byte_count_basis,
• LUT construction source / method,
• token_count,
• byte_count,
• bytes_per_token,
• whether the leading-space byte is counted in the LUT, the eval accumulator, or both,
• a val_loss / val_bpb sanity ratio.

That would not replace review or a semantic checker, but it would make this class of byte-accounting bug easier to catch consistently.

[riccardoalberghi]

This has been a known issue for weeks #897

[dexhunter]

Update — PR #1791 (genji0306, GDN K_KVShare_Wider FLA) shows the same 1.177× inflation fingerprint with a new wrinkle: the shipped helper code matches canonical (commit 14eae0a "fix: correct SentencePiece byte-accounting LUT to match base repo"), but the reported numbers were generated by the pre-fix eval path. The val_loss/val_bpb ratio diagnostic from the original issue still catches it cleanly:

seed	val_loss	val_bpb (claimed)	bytes/tok	canonical bpb
42	3.1466	1.0345	4.386	1.21857
1337	3.1404	1.0325	4.386	1.21620
2025	3.1397	1.0322	4.386	1.21594
			mean	1.21690

So a stricter heuristic would also reject submissions where the shipped val_loss/val_bpb ratio doesn't match what the shipped LUT produces on the canonical val stream — catches both code-level bugs and bin/log mismatches from out-of-date eval runs.

[HKati]

This surfaces two distinct failure modes:

1. a byte-accounting defect in the shipped eval path, and
2. an artifact-consistency failure: the shipped code, shipped logs, and reported metrics are not mutually consistent under a single eval path.

That is a strong argument for a small review-side metric receipt alongside any semantic checker. For example:

• eval path / commit fingerprint
• LUT construction source
• token_count / byte_count / bytes_per_token
• val_loss / val_bpb sanity ratio
• confirmation that shipped code, shipped logs, and reported metrics were produced by the same eval path

That would make this class of metric inconsistency easier to detect during review, not only reconstruct after the fact.