validator: isolate hidden-eval in a subprocess so a CUDA fault can't crash the validator

[gho11123]

Problem (observed in production)

A validator core-dumped while scoring a submitted bundle:

[op4] S3 forward — loading checkpoint.pt (124.2M params) → cuda:0
CUDA error: an illegal memory access was encountered
terminate called after throwing an instance of 'c10::Error'
Fatal Python error: Aborted (core dumped)

Call path: run_epoch → (score) → run_hidden_eval → compute_val_bpb → model.forward → F.linear.

The hidden-eval runs a submitted checkpoint's forward in-process. A fatal CUDA fault there is a C++ terminate() → SIGABRT, which Python cannot catch — so the entire validator process dies and judging halts until a manual restart. One submission is enough to take the validator down.

Fix — process isolation (reuse the pattern already in the tree)

_patched_hidden_eval already runs the eval in a subprocess via eval_in_workdir.py. This PR generalizes that to every checkpoint forward:

• Extract the spawn + result-parse into _run_eval_subprocess(workdir, ckpt_path, ralph_root, label).
• Route the canonical op4_hidden_eval path through it (workdir = RECIPE_DIR), not just the patched path.
• A child crash (non-zero exit, incl. a SIGABRT core dump) or timeout → (False, reason, None) → the bundle is rejected and the validator keeps running.
• The in-process CPU load_state_dict stays, so a structural-patch shape mismatch still routes to _patched_hidden_eval.

Behavior-preserving for the patched path (the label reproduces its exact messages); the only happy-path change is that the canonical forward now runs in a child.

Tests

tests/test_eval_subprocess_isolation.py: a non-zero child exit (core dump) and a timeout are caught as rejections; a valid RALPH_EVAL_RESULT line parses into a HiddenEvalResult.

🤖 Generated with Claude Code