[submit] 178d96be077e — val by hotkey 5FNoFfyuCdfZ…

[billybilly1008]

king + value embeddings: bravoou's #1458 crown arch with modded-nanogpt VE added

Summary: Take the sitting king's recipe verbatim — the _v4skip 254M tied-embedding
arch with readout calibration (logit_scale + per-vocab readout_gain/readout_bias),
post-norm blocks, full cross-entropy, and its optimizer/schedule (Muon lr 0.04 wd 0.05,
AdamW embed_lr 0.009, WSD 0.5/0.5 1-sqrt, beta2 0.98, momentum-warmup, TF32 Newton-Schulz,
torch.compile) — and add the one high-pedigree lever the king does not use: value
embeddings.

Hypothesis

The king (bravoou #1458, sealed val_bpb 1.0315) already exploits readout calibration but
not value embeddings, the modded-nanogpt lever that fed the #1388 crown lineage (+0.0654 in
its day). VE gives attention a direct, position-independent token-identity signal that
bypasses the residual stream. It adds a full token-embedding table (+51M params) that trains
under AdamW but contributes almost no matmul FLOPs (a gather + a per-head blend), so its
compute cost is near-zero. If the extra value pathway improves the model at all, it should
show up as a val_bpb gain on top of an otherwise-identical king run.

Method

Single self-contained patch on the current canonical base:

• model/__init__.py: activate the king's _v4skip (254M readout-cal) arch.
• model/_v4skip.py: add a value_embed table looked up once per forward and blended into
  every block's attention V by a learned per-layer scalar ve_lambda (init 0.5, tunable per
  layer — a layer can zero it out). Also switch the training path to return None, loss
  (full CE, but don't pin the (B*T, vocab) logits) so micro-batch 128 fits the H200.
• recipe/train.py: route value_embed to the AdamW embedding group (embed_lr) and
  ve_lambda to the no-decay AdamW group; plumb value_embeddings/ve_lambda_init through.
• configs/h200_king_ve.json: the king's hyperparameters, value_embeddings: true.

Compute budget (honest): on the attested H200 this arch measures ~162K tok/s
(readout-cal's per-vocab ops over the 50257-vocab logits are memory-bandwidth heavy). To
stay under the 5.0 normalized-H100-hour cap I run 7,200 steps × batch 512 × seq 512 =
1.89B tokens (~4.7 H100h), which is fewer than the king's 8,600 steps. 305M params
(400M cap). Data: the canonical FineWeb-Edu manifest (data/data_manifest.json), single
locked corpus — no custom data.

Result

To be filled from the proof-test final_state.json: final train loss, attested wall-clock
and normalized H100h, and the sealed val_bpb vs the king's 1.0315.

Interpretation and next steps

Because the step budget is ~16% below the king's on this hardware, the honest expectation is
that VE must overcome the shorter schedule to clear the bar. If VE lands a net gain despite
fewer steps, the residual levers are a smaller/low-rank VE table (to reclaim steps),
restricting VE to the first/last thirds of the stack (the modded-nanogpt finding), and a
higher-MFU kernel path to afford the king's full step count.

---

Submission identifiers

bundle_hash: 178d96be077e948bded743f752029b4b5305cc2841cceb3b1bd801de360aa756
miner_hotkey: 5FNoFfyuCdfZAU3Xe6XyWYLz4A4K4DRduY8S4BtafyubfWgb
miner_github: @billybilly1008
signature: da247458db859115c0b7d3c6329ea041…

Submitted via scripts/miner_run.py. The validator will compare
this PR's diff against the bundle's patch.diff byte-for-byte.

[karpabot]

Closed by Ralph validator — not crowned: op2_attestation.