This repository was archived by the owner on Aug 3, 2026. It is now read-only.
eval: fail closed when held-out shard/benchmark missing - #59
Merged
Conversation
bitzic
commented
Jun 25, 2026
bitzic
commented
Contributor
- run_hidden_eval silently fell back to random tokens (seed 424242) + placeholder benchmark when active_tokens.bin / active_benchmark.json were absent — scoring every checkpoint against noise
- that crowned the LEAST-trained model (val_bpb ~3.9, bench 1.0); the signature is live on sn40 (king gain ~0.0586)
- now fail closed by default; gate the synthetic fallback behind RALPH_ALLOW_SYNTHETIC_EVAL=1 (mirrors RALPH_ALLOW_MOCK_ATTESTATION)
- tests/conftest.py defaults the CPU suite to synthetic; fail-closed tests opt out via monkeypatch.delenv
- smoke_test: add synthetic + RALPH_REQUIRE_GH_PR=0 testnet relaxations
- new tests/test_eval_fail_closed.py; full suite 901 passed, 7 skipped
- run_hidden_eval silently fell back to random tokens (seed 424242) + placeholder benchmark when active_tokens.bin / active_benchmark.json were absent — scoring every checkpoint against noise - that crowned the LEAST-trained model (val_bpb ~3.9, bench 1.0); the signature is live on sn40 (king gain ~0.0586) - now fail closed by default; gate the synthetic fallback behind RALPH_ALLOW_SYNTHETIC_EVAL=1 (mirrors RALPH_ALLOW_MOCK_ATTESTATION) - tests/conftest.py defaults the CPU suite to synthetic; fail-closed tests opt out via monkeypatch.delenv - smoke_test: add synthetic + RALPH_REQUIRE_GH_PR=0 testnet relaxations - new tests/test_eval_fail_closed.py; full suite 901 passed, 7 skipped Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
rockmania99
pushed a commit
to rockmania99/ralph
that referenced
this pull request
Jun 26, 2026
…l_bpb The validator scores the miner's OWN forward() to compute val_bpb — op4 runs the patched model code for any structural patch — feeding the whole window in one call. The target for position t is input[t+1], which sits inside the model's input, so a non-causal forward can read the answer and emit a perfect prediction, collapsing val_bpb to ~0 and crowning an unbeatable, fraudulent king. This is not closed by the merged-PR gate (RalphLabsAI#60) or eval fail-closed (RalphLabsAI#59): the cheat carries a real merged recipe PR and a real attestation. Fix: - assert_causal(): before trusting val_bpb, probe the model — overwrite the FUTURE positions of a real eval window with a DIFFERENT real held-out slice and require the earlier logits to be unchanged. A causal model is invariant; a look-ahead forward is not. A realistic (not uniform-random) decoy future closes the adaptive-probe evasion. Runs inside run_hidden_eval, covering both the canonical and patched-eval paths. - Pin the eval window validator-side (EVAL_SEQ_LEN), capped at the model's max_seq_len, instead of the miner-controlled cfg.max_seq_len // 2 — otherwise a miner can enlarge the eval context to score an easier eval than the king. - tests/test_eval_forward_trust.py: honest model passes, look-ahead rejected (directly and via run_hidden_eval), short-stream no-op, pinned constant. Follow-up (see PR description): the related eval-file-read variant — a forward that reads the held-out shard off disk inside the patched-eval subprocess — needs OS sandboxing of that subprocess; the causality probe does not cover it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.