Record: PR #1787 base + PPM-D OMP byte mixture (val_bpb 1.0322 3-seed mean)

[dexhunter]

Summary

PR #1787 (nprime06) base + Smear gate (BOS-fixed) + LQER asymmetric rank-4 + PPM-D order-4 byte-level mixture (port of the PPM-D class from PR #1850, rewritten in C and parallelized with OpenMP). Strictly score-first byte-by-byte: PPM context tables are updated after each byte is scored.

• 3-seed mean: val_bpb = 1.0322 (std 0.00064) — seeds 314, 42, 1234
• val_loss: 0.7155 nats/byte (std 0.00045)
• Beats merged SOTA PR Record: SP8192 + 3-Layer Recurrence + Parallel Residuals + QK-Gain 5.25 + Legal TTT — val_bpb 1.0810 (3-seed mean) #1493 (1.0810) by ~0.05 BPB

Results (8×H100 80GB SXM, 600s train, no neural TTT)

Seed	Steps	ms/step	Post-EMA BPB	Post-PPM BPB	val_loss (nats/byte)	Artifact (bytes)
314	4658	128.0	1.07320	1.03191	0.71526	15,996,077
42	4679	127.4	1.07231	1.03176	0.71516	15,995,309
1234	4675	127.5	1.07354	1.03294	0.71598	15,998,552
Mean	4671	127.6	1.07301	1.03220	0.71547	15,996,646
Std			0.00065	0.00064	0.00045

All 3 seeds clear: artifact ≤ 16,000,000 bytes (max 15,998,552), train_time ≤ 600s (max 596.09s), eval_time ≤ 600s (max 297.9s).

Mechanism summary

1. PR Record: PR #1736 + Polar Express NS + MIN_LR + Sparse Attn Gate + Fused CE + PR #1767 TTT — val_bpb 1.06335 #1787 base stack — SparseAttnGate + PolarNS + MIN_LR + FusedCE.
2. Smear gate (window=12, BOS-masked) — content-conditioned 1-token causal lookback on first 12 residual dims; smear is reset to zero at every document boundary.
3. LQER asymmetric rank-4 — inline post-GPTQ low-rank residual correction on top-3 weight tensors (group=64).
4. PPM-D byte mixture (PPM_NATIVE_ENABLED=1, order=4) — byte-level Markov contexts of orders 0..4 with escape-D smoothing, mixed with NN per-byte logits as p_mix = (1−λ)·p_NN + λ·p_PPM, where λ adapts between λ_hi=0.9 / λ_lo=0.05 based on PPM context confidence (threshold=0.9). The PPM-D table is updated after scoring each byte (strictly score-before-update). OpenMP parallelization across 8 chunks of 4M tokens reduces PPM scoring wall-time from ~957s baseline to ~95–190s, fitting the 600s eval budget.

Issue #1017 compliance

• Cond 1 (causal) — model uses causal attention; PPM tables are updated byte-by-byte AFTER scoring.
• Cond 2 (full normalized distribution) — both NN logits (softmax over 8192 tokens) and PPM-D byte distribution (over 256 bytes with escape-D smoothing) are individually normalized; their convex mixture is too.
• Cond 3 (score-before-update) — for each byte b_i: compute p_mix(b_i | context), accumulate −log p_mix(b_i), then update the PPM tables. See score_byte() in the embedded C source inside train_gpt.py.
• Cond 4 (single L→R pass) — no rescore/selection/reordering. Note: a TTT length-sort batching helper is present in the source for code-path completeness but is not called at eval time when TTT_ENABLED=0 (the active path is gated by ppm_only_path = h.ppm_native_enabled and not h.ttt_enabled and goes directly to run_ppm_native_pass).
• Section V — byte-level BPB via SentencePiece piece table with PR Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019 +1 boundary credit; full 40,540,160 validation tokens / 151,078,222 bytes scored, no subset.

Lineage

• Builds on PR Record: PR #1736 + Polar Express NS + MIN_LR + Sparse Attn Gate + Fused CE + PR #1767 TTT — val_bpb 1.06335 #1787 (nprime06, base) + PR Record: SP8192 + Strict Full-Val Byte PPM Mixture — 1.00495 BPB (3-seed mean) #1850 (someone114514, PPM-D mechanism class).
• Stack also incorporates PR Record: LeakyReLU² + Legal Score-First TTT + Parallel Muon — val_bpb 1.1194 (3-seed mean) #549 / PR Record: AR Self-Gen GPTQ + XSA-all + BigramHash 3072×112 — val_bpb 1.11473 (3-seed mean) #1019 / PR Record: SP8192 + GPTQ Embeddings + Depth Recurrence + MuonEq-R + SDClip — val_bpb 1.08563 (5 seed mean) #1394 / PR Record: Varlen attention + fused MLP + doc-independent TTT (1.07336) #1530 lineage as documented in the submission's README.md.

Reproducibility

• Self-contained train_gpt.py (183,428 bytes); all mechanism flags via env vars in the README's Run command.
• Requires gcc with OpenMP support on the eval host (standard on all Linux distros).
• PPM-D native C source is embedded as a string literal inside train_gpt.py and compiled at eval time via subprocess('gcc -O3 -march=native -fopenmp'). No external network calls during eval.

Test plan

• 3-seed BPB mean / std verified across seeds 314, 42, 1234
• All 3 seeds pass artifact ≤ 16,000,000 bytes (decimal)
• All 3 seeds pass train_time ≤ 600s
• All 3 seeds pass eval_time ≤ 600s (max 297.9s)
• Issue A Field Guide to Valid Submissions #1017 Conditions 1-4 + Section V audit per submission README
• No external network during eval (PPM compiled from embedded C source)
• Python 3.10 compile check (no PEP 701 nested f-strings)
• Seed logs scrubbed of absolute working-directory paths

[dexhunter]

Closing this PR — on reflection, the timing doesn't work for chronological priority.

PR #1850 (someone114514) filed several hours earlier with the same PPM-D byte mixture mechanism class at a stronger BPB (1.00495). Per the chronological-priority review process, if the PPM-D class is approved, PR #1850 takes precedence and our 1.0322 wouldn't clear the merge bar over their submission. If the class is ruled out, neither merges.

The interesting work in this submission (gcc-OpenMP parallelization of the PPM-D scoring pass: ~957s → ~190s, making full-val byte-level scoring fit the 600s eval budget) is happy to live in the discussion thread on PR #1850 if useful to that author or anyone else iterating on the class.

Apologies for the noise on the review queue.