This repository was archived by the owner on Aug 3, 2026. It is now read-only.
Fix/op4 hosb wiring - #79
Merged
Merged
Conversation
- RALPH_HOSB mode: off (default, legacy, byte-unchanged) | shadow (legacy
crowns, HOSB logged to hosb_shadow.jsonl for king calibration) | enforce
- LEAK-FREE sandbox protocol: container gets ONLY idx_grid + scored_idx and
emits TOP-K logits; HOST computes CE (ce_from_topk_logits) against private
targets. tgt_grid NEVER mounted; V host-pinned from the checkpoint config,
not the container manifest; emitted top-K validated (shape/finite/in-range/
no-dup/logsumexp>=lse_topk)
- op4_hidden_eval(...,chain): legacy body -> _legacy_hidden_eval; HOSB dispatch;
judge_submission threads chain; block-hash epoch seed; honest entropy docstring
- ENFORCE GUARD: enforce fail-closes (rejects, never crowns) unless
RALPH_HOSB_ENFORCE_ACK=1, because 2 verified container-side forgeries remain
open and gate enforce (3 red-team passes):
1. top-K logsumexp/partition-function deflation (host can't verify the
softmax denominator from top-K alone; ~0.02-0.04 bpb, crown-relevant)
2. benchmark_accuracy still container-reported (Branch C crown; pre-existing
parity with the legacy sandbox)
-> Stage 4 (enforce hardening): host-chosen-index emission with an importance-
sampled partition function + host-reduced benchmark, then drop the guard
- off-mode proven byte-unchanged; enforce proven fail-closed; safe to merge
DEFAULT-OFF + use shadow for on-king tolerance calibration
- 17 CPU tests
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- design+red-team vetted (workflow wb83ti096): the seed importance-decoy estimator was proven BROKEN (decoy suppression deflates ~0.1 bpb); the winner is the host-built lower bound - ce_from_topk_logits: DROP the container logsumexp param entirely. Host computes Z_hat = logsumexp(emitted top-K) itself — a true lower bound on the full-vocab partition function. CE = Z_hat - logit[target] (hit) / Z_hat - min(top-K) (miss). Container can't shrink Z below its own top-K, and CE is uniform-shift invariant → the "claim tail=0 to deflate" and rescale forgeries are structurally impossible - container (sandbox_eval --grid) stops emitting logsumexp.npy; host loads only topk_logits/topk_indices; RALPH_HOSB_TOPK default raised 256 -> 4096 so the residual -log(1-tail_K) deflation stays under the crown margin (degenerate emission that drops mid-rank tokens self-defeats via huge miss CE) - V (checkpoint config) now used ONLY for index-range validation; no longer in the CE math, so the manifest-vocab forgery is structurally gone too - enforce guard message updated; guard STAYS (stage-4b benchmark host-reduce + king re-score + on-box calibration still gate dropping it) - tests: host-owns-partition (no logsumexp), uniform-shift invariance, degenerate-miss self-defeat, leak-free roundtrip via lower-bound-Z Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- closes the second enforce forgery (container-reported benchmark_accuracy crowning via scoring Branch C with zero quality gain) - make_placeholder_examples: target + distractors drawn from ONE exchangeable pool (content whitening) so no monotone token-id function distinguishes the answer (the "pick smallest id" cheat scored 78.5% on the deployed file) - build_benchmark_grid: host shuffles each example's candidates with a PRIVATE per-example permutation, emits only context + shuffled candidate ids (no correct-index marker); correct_pos stays host-side - reduce_benchmark_scores: host argmax vs the private correct slot, fractional tie credit (all-equal/blind → exactly 1/C); argmax is permutation-equivariant → bit-identical to the legacy scorer for an honest model (king comparable) - sandbox_eval --grid: stop computing benchmark in-container / mounting the answer key; emit a per-shuffled-candidate score matrix the host reduces - _hosb_sandbox_nlls: build+mount the shuffled benchmark grid (no answers), host-reduce; benchmark_accuracy is now HOST-computed, never the container - enforce guard updated (both axes host-owned; king re-score + on-box calibration still gate dropping it) - tests: whitening exchangeable, no-marker grid, HRB==legacy honest, blind/all-equal/smallest-id bounded at chance, no-answer-key-mounted e2e Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Final red-team (wu77vmspe) found a CRITICAL latent since merged stage 2: the blanked grid scored MANY positions per window, each blanking only its own suffix, so a sibling row scored at e'>=e+1 left window[e+1] un-blanked — the container (handed the full idx_grid) recovered ~93% of answers off siblings and forged val_bpb 0.17-0.25 past both witnesses. (Partition fn + benchmark were confirmed sound/host-owned; this is the one true hole.) - CRITICAL: build_blanked_grid now scores EXACTLY ONE position per window; witness rows (A/B, wrong-target) all blank the SAME suffix, and windows are disjoint slices → no row exposes another's answer. + a defensive invariant assert and a reproduction test (sibling-recovery now 0%, was 93.5%). n_scored_per_window retained for API compat but ignored. - HIGH: ce_from_topk_logits computes CE from MAX-SHIFTED logits (cancellation- free) and rejects |logit| > 1e4 — kills the all-huge-logit z_hat-logit→0 forgery. - MEDIUM: filler is now a DISJOINT seeded-random corpus, not the secret shard (a causal model ignores input[>e], so the filler value is irrelevant; the old shard-slice filler was a 2nd answer-recovery path + shard-secrecy erosion). - MEDIUM: fail-closed if the wrong-target witness quota is empty on a real grid. - LOW: build_blanked_grid moved inside the fail-closed try (short shard → reject, not crash); corrected the V/RALPH_VOCAB_SIZE comment. - guard STILL kept; king re-score (+ a king-record scoring-path tag) and on-box calibration remain the gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-verify (w2f4gxc3z) confirmed the within-window leak is closed but found a distinct cross-WINDOW boundary leak: scoring e=L-1 makes the answer window[L] == the NEXT window's un-blanked input column 0, recoverable + one-hottable to CE=0 past both witnesses. Sub-margin at L=512 but a miner can shrink max_seq_len (L=16 -> ~5% boundary -> ~0.08 bpb deflation, ~6x the crown margin); king re-score does NOT close it (asymmetric). - build_blanked_grid never scores e=L-1 (cap e <= L-2) so every answer window[e+1] stays inside THIS window's blanked region. Verified: 0 boundary cells, 0 cross-window recovery (was the open hole). - ce_from_topk_logits: _MAX_ABS_LOGIT 1e4 -> 100 (honest logits/softcap ~30-50); the 1e4 target/-1e4 rest emission that rounded CE to exactly 0 is now rejected. - wrong-target witness threshold 0.5 -> 0.1 (an honest LM's sub-floor frac on random distractors is ~0; 0.5 was far too loose). - tests: assert no e=L-1 cell; huge-logit rejected; collision-tolerance at 5%. - REMAINING (documented, guard kept): king re-score as a hard enforce precondition + equal val_seq_len gate + a max_seq_len lower bound + noise_floor >= 3x measured HOSB std, all settled by on-box calibration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.