Record: SP8192 + Sliding-Window Eval + Conditional-PPM Byte Mixer Full-Val - val_bpb 1.015784

[anmarhindi]

Summary

3-seed mean val_bpb = 1.015784 (std 0.000524), an improvement of −0.065 BPB over the current leaderboard SOTA (1.0810). 8×H100 SXM, training ≤600s, eval ≤600s, total submission 15,576,768 bytes (≤16,000,000).

The contribution is a single eval-time post-processor on top of the PR #1493 stack: a conditional byte-level PPM mixer that derives P_NN(byte_0 | history) from the model's SP8192 softmax via a canonical first-byte LUT, then mixes it with a PPM-D byte conditional through a per-byte sigmoid gate. No new trainable parameters, no additional artifact bytes, no training-time changes.

Seed	val_bpb
42	1.01519
1337	1.01596
314	1.01620
Mean	1.015784
Std	0.000524

Each seed's eval runs over the full validation set (9,662,464 tokens; 32,756,252 canonical bytes), gathered across all 8 ranks before the byte-level PPM-D state advances. submission.json records eval_full_val_verified: true, eval_token_count_per_seed: 9662464, and eval_canonical_byte_count_per_seed: 32756252 as forensic attestations.

A std of 0.000524 puts the −0.005-nat threshold at p ≪ 0.01.

Relationship to PR #1835

PR #1835 introduced byte-level mixing using the approximation:

P_NN(byte = b) ≈ exp(token_logp / n_bytes_in_token)

This value is identical for every byte in the realized token and does not sum to 1 over the byte alphabet — it is a per-byte-position constant rather than a probability distribution. The C2 concern raised on the #1835 thread follows directly: scoring the same realized byte yields different probabilities depending on which token later turns out to be correct, which is not autoregressive at the byte alphabet.

This submission instead derives P_NN(byte_0) as a proper marginal over the SP8192 alphabet, weighted by a canonical first-byte LUT (rebuilt from the SP tokenizer at deserialize, no extra storage):

P_NN(byte_0 = b | history)
    = Σ_{T : canonical_first_byte(T, prev) = b}  P_NN(T | history)

The "canonical first byte" of token T at position t depends on whether the previous token x[t] is a boundary: if T carries the SP-BPE leading-space marker AND prev is non-boundary, the realized canonical first byte is 0x20; otherwise it is the SP-piece's natural first byte. ValidationData.__init__ builds two first-byte masks (mask_no_space for prev-IS-boundary, mask_with_space for prev-non-boundary), and eval_val_sliding selects per-position via is_boundary_token_lut[prev]. This eliminates the per-token uniform-split fallback that earlier drafts of this mixer used for the include_space case, which would have re-introduced the C2 issue from #1835 for the ~40% of positions where include_space fires.

For the remaining bytes within the realized token (b₁..b_{k−1}), the chain-rule residual is:

P_NN_rem(b₁..b_{k−1} | b₀, history)
    = P_NN(token | history) / P_NN(byte_0 | history)

Both terms are proper distributions over their respective alphabets:

• byte_0: convex combination over the 256-byte alphabet.
• remainder: convex combination over the joint byte-sequence alphabet of length k−1.

Their product is a proper distribution over the realized token's byte stream, so C2 holds by construction at every position regardless of leading-space status.

The mix gate is a sigmoid on PPM context confidence:

λ = 1 − sigmoid(α · (conf_PPM − β))    α = 15.0, β = 0.80
P_mix(b) = λ · P_NN(b) + (1 − λ) · P_PPM(b)

Why this works

The 16 MB cap forces a 35M-parameter model that cannot allocate enough mass to the long tail of byte sequences (URLs, code identifiers, repeated proper nouns, numerical literals). PPM's strength is exactly that tail: order-5 byte context routinely assigns near-1 probability to the next byte inside a code block or a recurring named entity, while a parameter-constrained NN has to spread mass thin. The sigmoid gate captures this conditionally — trust PPM when its top-symbol probability is high, fall back to the NN otherwise. Deriving P_NN(byte_0) as a proper marginal of the SP8192 softmax (rather than a per-byte uniform-split approximation) means the gate weights two distributions that actually live on the same alphabet, so C2 holds at every position regardless of whether a leading-space prepend applies.

Ablation

3-seed means, applied sequentially:

Stage	val_bpb
pre-EMA, pre-quant	1.0876
post-quant (no mixer)	1.1017
sliding-window stride-64 (PR #1493)	1.1067
+ cond-PPM mixer (full val)	1.0158

The mixer contributes −0.091 BPB at eval time over the full validation set (post-sliding-window → post-cond-PPM). Local Blackwell ablation predicted larger absolute mixer gains on weaker base models — where the long-tail miss is more severe — so the better-trained base here narrows the absolute headroom while still landing well below SOTA.

Compliance (Issue #1017)

Condition	How this submission satisfies it
C1, causality	Sliding-window scoring is strict-past only. Cond-PPM byte state advances only after each byte's mix log-prob is recorded. Marginalization at byte_0 derives from the position's softmax (which sees only strict past) using the appropriate first-byte mask conditional on is_boundary_token_lut[prev]. Mix gate weights depend on PPM context confidence only, never on the realized byte.
C2, normalization	byte_0 mix is between two byte-alphabet distributions (model marginal vs PPM-D byte conditional); the remainder mix is between two joint-byte-sequence distributions over the realized token's byte-1..k-1 alphabet. Both are proper. No per-token uniform-split approximation, no fallback path.
C3, score-first	Both NN softmax and PPM byte conditional commit before observing the realized byte at each step. PPM state advances post-scoring.
C4, single L→R pass	Each validation token contributes exactly one BPB term. Sliding-window scoring overlaps, but each token is scored at exactly one position.

No SLOT, no n-gram cache outside the legal byte-level PPM-D state, no logit bias, no ETLB, no pre-quant TTT (which would violate C3). Standard softmax over SP8192 at every scored position.

Eval-set coverage. The cond-PPM mixer runs over the full validation set (9,662,464 tokens / 32,756,252 canonical bytes per seed), with chunks gathered across all 8 ranks before byte-level PPM-D state advancement on rank 0. This is recorded in submission.json as eval_full_val_verified: true, eval_token_count_per_seed: 9662464, and eval_canonical_byte_count_per_seed: 32756252. Each seed's train_seed*.log contains a cond_ppm tokens=9662464 bytes=32756252 cond_mix_bpb=... line as forensic evidence.

Reproduction

Full reproduction commands, per-seed train + eval logs, and helper modules (CaseOps tokenizer + transform) are in records/track_10min_16mb/2026-04-30_CondPPM_1.015784/. The bundled train_gpt.py is lzma+base85 wrapped (49,485 bytes); the underlying source lives at obliterate_0p9/train_gpt.py on the cond-ppm branch of the author's repo.

git clone https://github.com/anmarhindi/parameter-golf
cd parameter-golf
git checkout cond-ppm
cd obliterate_0p9
bash run.sh                  # 3 seeds × (≤600s train + ≤600s eval)
bash build_submissions.sh    # produces submission folder + tarball

To inspect the readable source of train_gpt.py without executing it:

import lzma, base64, re
src = open("train_gpt.py").read()
blob = re.search(r'b85decode\("([^"]+)"\)', src).group(1)
print(lzma.decompress(base64.b85decode(blob)).decode())

Artifact size

Component	Bytes
final_model.int6.ptz (int6 GPTQ + brotli)	15,527,283
train_gpt.py (lzma+base85 wrapped)	49,485
Total counted toward 16 MB cap	15,576,768
Headroom	423,232 (2.6%)

Helper modules (lossless_caps.py, prepare_caseops_data.py) and the SP8192 tokenizer are included for reproduction but do not count toward the cap, per Issue #1017 §III.

Acknowledgments

This submission stands on a chain of prior work: @clarkkev (PR #1394) for the SP8192 base; @samacqua (PR #1530) and @romeerp (PR #1729) for CaseOps; @nprime06 (PR #1787) for parallel residuals + looping; @dexhunter (PR #1797) for Smear + LQER; @cocohearts for the SmearGate BOS-mask fix; and the sliding-window stride-64 evaluator from PR #1493. The PPM-D byte conditional itself is classical (Cleary & Witten 1984; Moffat 1990; Howard 1993). What's contributed here is the per-position canonical first-byte marginalization that closes the C2 gap raised on PR #1835's earlier byte-mix attempt, and the empirical observation that an order-5 byte conditional pairs well with a parameter-constrained LM at this scale.

[leon2k2k2k]

This block of code breaks C1 (because include_space uses the current token to choose the probability distribution) and C2 (the flat fallback).

b0 = token_bytes[0]
  ppm_log_b0, conf_b0 = _ppm_byte_logprob_and_conf(b0, window, ctx_counts)                                       
  nn_p_b0 = float(byte0_nn_prob[i])              # ← the proper marginalization
  if include_space:                                                                                              
      try:                                                                                                       
          nn_p_b0 = math.exp(-float(nll_nats[i]) / n_bytes)    # ← FLAT FALLBACK                                 
      except OverflowError:                                                                                      
          nn_p_b0 = 0.0                                                                  

[anmarhindi]

This block of code breaks C1 (because include_space uses the current token to choose the probability distribution) and C2 (the flat fallback).

b0 = token_bytes[0]
  ppm_log_b0, conf_b0 = _ppm_byte_logprob_and_conf(b0, window, ctx_counts)                                       
  nn_p_b0 = float(byte0_nn_prob[i])              # ← the proper marginalization
  if include_space:                                                                                              
      try:                                                                                                       
          nn_p_b0 = math.exp(-float(nll_nats[i]) / n_bytes)    # ← FLAT FALLBACK                                 
      except OverflowError:                                                                                      
          nn_p_b0 = 0.0                                                                  

Good catch on the if include_space branch. That was a legacy code path that should have been removed before packaging. The cond-PPM construction itself (proper marginal of the SP8192 softmax via the canonical first-byte LUT) was always the intended contribution; the fallback was a stray remnant from earlier scaffolding and shouldn't have been live.

Removed it. ValidationData.__init__ now builds two first-byte masks (mask_no_space for prev-IS-boundary, mask_with_space for prev-non-boundary), eval_val_sliding selects per position via is_boundary_token_lut[prev], and _cond_ppm_mixture_bpb uses byte0_nn_prob[i] directly with no fallback path. Every position scores byte_0 against a proper byte-alphabet marginal regardless of leading-space status, fully C1/C2 compliant, no formula branching on the realized token.

Re-ran 3 fresh seeds end-to-end with the corrected code as proof. val_bpb = 1.015784 (std 0.000524), full val (9,662,464 tokens / 32,756,252 canonical bytes per seed). Per-seed: 1.01519 / 1.01596 / 1.01620. Each train_seed*.log has the cond_ppm tokens=9662464 bytes=32756252 line as evidence. The current PR head should be reviewable as-is.

[cocohearts]

Leaderboard audit note (pre-cutoff state): I don't think this is valid as a record row. The include_space C1/C2 fallback was fixed before cutoff, but the reported cond-PPM BPB still appears to use the wrong scoring surface/denominator: the logs score 9,662,464 tokens / 32,756,252 reconstructed token bytes, while the standard eval for that same split implies about 29.95M raw validation bytes. The official CaseOps BPB denominator should be the raw validation byte sidecar, not reconstructed transformed token-piece bytes; correcting seed 42's denominator puts the cond-PPM score around 1.11 BPB, not 1.015. It also is not the full current 50k-doc CaseOps validation set used by the accepted rows.