This repository was archived by the owner on Aug 3, 2026. It is now read-only.
Feat/validator sandbox - #71
Merged
Merged
Conversation
- validator/sandbox.py: build_docker_argv (network none, cap-drop ALL, read-only, non-root, no-new-privs, ipc private, pids/mem/cpu caps, single pinned GPU, digest-pinned image, no secret mounts), preflight (fail-closed: docker/runc/toolkit version floors + userns-remap active + cuda-compat hook-disabled + image present), run_in_sandbox with a docker-kill watchdog, parse_eval_line (strict-typed + quantized output) - env is allowlist-only, blocklist-asserted (reuses proof.runner) - SandboxUnavailable => callers must reject, never fall back to bare exec - inert until op4/audit call sites are routed through it (next) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- eval/host_reduce.py: reduce per-position NLLs (emitted by the sandboxed miner forward) into bpb on the HOST — host owns the formula, token count, bytes_per_token, tail mask, and eval-set hash - replaces the miner-printed scalar + 100-token hash for the structural-patch path; rejects wrong-length / non-finite / negative arrays (fail-closed) - equivalence-tested against eval.val_bpb.compute_val_bpb - imported by nothing yet => vali behavior UNCHANGED, still runnable - anti-forgery teeth still require the tightened independent audit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- validator/sandbox_eval.py: in-container entrypoint. prepare_workdir (copytree canon + apply_patch, contained in-container) then run_sandbox_eval emits per-position NLLs (np), NOT a reduced score; the HOST reduces them via eval.host_reduce. Imports trusted helpers before placing the patched workdir on sys.path. - eval/val_bpb.py: add per_position_nlls (additive; same windowing as compute_val_bpb, equivalence-tested) - Dockerfile.sandbox: cuda-RUNTIME, no git/compilers, eval/private NOT baked (dockerignore), non-root, digest-pinned - tests: end-to-end CPU produce->reduce == compute_val_bpb - still inert (no RALPH_SANDBOX wiring) => vali behavior UNCHANGED, runnable Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- validator/sandbox.py: drop unused os + dataclasses.field imports (F401) - tests/test_host_reduce.py: split semicolon statement (E702) - tests/test_sandbox.py: wrap long stdout literal (E501) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes sandbox mode: when RALPH_SANDBOX=1, op4_hidden_eval runs the miner's (possibly patched) model inside the hardened container and the HOST reduces the crown-critical val_bpb from the emitted per-position NLLs. - sandbox_eval.py: entrypoint now (canon, patch, ckpt, eval_dir, out) — prepare_workdir (patch applied in-container) then emit nlls.npy + benchmark accuracy (compute_benchmark_score, cheap) in manifest.json - validator.py: _sandboxed_hidden_eval — preflight FAIL-CLOSED (SandboxUnavailable => reject, never bare-exec fallback), digest-pinned RALPH_SANDBOX_IMAGE, RECIPE_DIR/proof_dir/eval-private mounted ro, host computes eval_set_hash + host_reduce(nlls) -> val_bpb; op4 routes here behind RALPH_SANDBOX (default off) - Dockerfile.sandbox: run-contract comment matches the 5-arg entrypoint - tests: op4->sandbox->host-reduce == in-process val_bpb; fail-closed on SandboxUnavailable; sandbox_eval now takes eval_dir Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve validator.py op4_hidden_eval conflict: keep BOTH the hidden-eval result cache (main, PR#66) and the sandbox route (this branch). Cache check runs first (benefits both paths); the sandbox route now also caches its result so a deferred challenger isn't re-containerised every epoch.
- _sandboxed_hidden_eval: wrap in try/finally so the per-submission /out host scratch (nlls.npy + manifest, ~12 MB) is removed on every exit path (success, fail-closed, reject) — no /tmp accumulation. Container teardown was already automatic via docker --rm. - test asserts no ralph_sbx_out_* leak after op4 - document the ephemeral-VRAM residual (eval set is mounted per-submission anyway, so residual VRAM adds no new exposure; driver/MIG is the real mitigation) rather than bolt on an in-band scrub Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.