fix(tbench2_env): keep verifier assets out of the agent's reach

[nblintao]

Summary

Both execution modes let the agent read — and tamper with — the assets the verifier scores against: tests/test_outputs.py spells out the expected outputs, and solution/ is the literal answer. That's fine for evaluation plumbing with a cooperative agent; for RL training it is a reward-hacking vector. This PR aligns both modes with the official TB2 harness's model: the solution never ships, and tests exist in the agent's filesystem only during the verify window.

• Local mode: with TB2_WITHHOLD_TESTS=1, reset() reads tests/ into server memory and deletes it (and solution/ — nothing in this env reads it; it exists only for oracle runs) from disk before the agent's first action. Gated off by default because in plain local mode TB2_TASKS_DIR may be a developer's working checkout that must not be modified; deployments that stage per-task copies (e.g. cloud sandboxes — see the stacked feat(tbench2_env): per-task sandbox image recipe + Daytona materialization #966) turn it on.
• Staging hardening (unconditional): evaluation now recreates both official fixed paths (/tests, /logs/verifier) from scratch before scoring. The previous mkdir -p && cp -a merged into whatever already existed, so an agent could pre-plant a /tests/conftest.py (pytest auto-loads it) or a symlinked /logs/verifier and have it survive into scoring. The bare-pytest fallback now runs against the same staged copy.
• Docker mode gets true stage-at-verify: the initial /task copy excludes tests/ and solution/, and _evaluate_docker streams tests in from the host — this server sits outside the container, so the staged copy is one the agent never saw — right before pytest runs, and removes them again after scoring (in a finally: docker-mode step() reports a scoring error without ending the episode, so an errored verify must not leave /task/tests behind for the agent's still-live session).

Residual risk, shared with the official TB2 harness: verification necessarily runs inside the same container the agent controlled, and agent-started background processes must be left alive (many tasks are scored on services the agent set up), so a leftover watcher that copies the staged tests or rewrites the reward file during the verify window — or a root agent that tampers with container binaries — could still fake a pass. Task state (services, git state) is not portable, so verification cannot move to a container the agent never touched.

Stacked on #965. This branch is based on #965's, so its commit appears here until it merges — review this PR's own commit, fix(tbench2_env): keep verifier assets out of the agent's reach (the last commit). The per-task sandbox recipe (#966) now stacks on top of this PR and enables the gate in its server command. (GitHub can't retarget a fork PR's base onto another fork branch, so the base reads main.)

Type of Change

• Bug fix

Alignment Checklist

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles (verifier integrity is a precondition for "one env = one trajectory" reward semantics)
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated (no API changes; one new opt-in env var, documented in the README)
• I have run bash .claude/hooks/lint.sh — the changed files are clean (pre-existing failures in other envs are untouched)

RFC Status

• Not required (bug fix, docs, minor refactoring)

Test Plan

• In-repo pytest (tests/envs/test_tbench2_env.py, runs in CI, no camel/docker/daytona needed): withhold round-trip (disk cleared, memory populated, task.toml/instruction.md untouched, idempotent re-reset); staging wipes a pre-planted conftest.py and stale reward.txt; without the gate the source checkout is never touched; full evaluate flow scores from the in-memory copy via the canonical harness; docker mode's copy exclusions, no-duplicate archive, evaluate-time staging order (rm -rf → fresh copy → pytest → rm -rf), and cleanup when scoring itself raises, via a mock container.
• End-to-end on a per-task Daytona cloud sandbox (fix-git, gate enabled): the staged task dir carries no tests/ or solution/; a filesystem-wide find turns up neither anywhere agent-readable; evaluate on the untouched task scores 0.0 via the canonical tests/test.sh run from the in-memory copy; applying the official oracle solution then scores 1.0.
• Suite-wide static scan: all 89 official tasks' tests/ trees contain only regular files and directories (no symlinks or special modes), so the tar round-trip is safe across the whole benchmark.

---

Note

Medium Risk
Changes reward-critical verify/staging paths and can delete task tests//solution/ when TB2_WITHHOLD_TESTS=1; misconfiguration on a shared checkout or multi-worker server could break scoring or corrupt tasks.

Overview
Stops agents from reading or tampering with tests/ and solution/ during episodes by aligning local and Docker TB2 execution with official harness behavior: verifier material exists on the agent filesystem only for the scoring window.

Local mode: Optional TB2_WITHHOLD_TESTS=1 (or withhold_tests=True) on reset() tarballs tests/ into server memory, deletes tests/ and solution/ from the task dir, and restores tests at /tests only inside evaluate. Docker mode: Initial /task copy skips tests/ and solution/; _evaluate_docker wipes /task/tests, streams tests from the host, runs pytest, then removes them in finally.

Scoring hardening (always on locally): Evaluation recreates /tests and /logs/verifier via _hard_remove (symlink-safe), runs canonical test.sh or a staged fallback pytest path under a lock, writes harness logs under /logs/verifier instead of /tmp, and always tears down staged paths after verify—even on partial failures. Docs add the new env var; tests/envs/test_tbench2_env.py adds broad unit coverage for withholding, staging, symlinks, and mock Docker verify flows.

^{Reviewed by Cursor Bugbot for commit 953aad3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 953aad3. Configure here.}

[cursor]

Stale withheld tests cache

Low Severity

_stage_tests_for_verify always restores from _WITHHELD_TESTS when a class-level entry exists, without checking self.withhold_tests or whether on-disk tests/ was updated. After a prior withheld reset in the same process, scoring can use an outdated tarball even when tests/ was restored or edited on disk, because _withhold_verifier_assets skips re-tarballing when the key is already cached.

Additional Locations (1)

• envs/tbench2_env/server/tbench2_env_environment.py#L445-L454

 

^{Reviewed by Cursor Bugbot for commit 953aad3. Configure here.}