fix(tbench2_env): canonical test.sh scoring, real timeouts, task-image workdir#965
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
289b100 to
4745fd2
Compare
501a20c to
82859ba
Compare
…ask env & workdir - _evaluate_task runs the task's official tests/test.sh (pinned uvx toolchain, /logs/verifier/reward.txt) when the task ships it; the bare pytest fallback (custom task dirs without the canonical harness) prefers uvx with a pinned pytest so it carries its own toolchain, and runs from the same workdir the agent worked in. - Pass timeouts through to every shell_exec call: camel's shell_exec has its own per-call default (20s) and ignores the constructor timeout, so agent commands and evaluations were silently truncated at 20s (circuit-fibsqrt's verifier declares 3600s and was cut mid-test). Agent execs use command_timeout_s (TB2_COMMAND_TIMEOUT_S-overridable); evaluate uses the task's own [verifier].timeout_sec budget. - Shell cwd resolves to the task image's own WORKDIR, parsed from the task's environment/Dockerfile (TB2_TASK_WORKDIR overrides): task images pin their own WORKDIR (fix-git: /app/personal-site, prove-plus-comm: /workspace) and both agents and oracle solutions assume commands run there. Falls back to the task source dir for plain local mode. - Drop install_dependencies=['pytest'] (evaluation carries its own pytest via uvx); with CAMEL_RUNTIME=true camel's .initial_env venv then stops hijacking PATH away from the task image's native python. - Serialize canonical evaluation under a lock so concurrent sessions on one server cannot clobber the official fixed verifier paths (/tests, /logs/verifier/reward.txt).
82859ba to
85343bd
Compare
| m = re.match(r"^\s*WORKDIR\s+(\S+)", line, re.IGNORECASE) | ||
| if m: | ||
| workdir = m.group(1) | ||
| return workdir |
There was a problem hiding this comment.
Multi-stage WORKDIR parsing is wrong
Medium Severity
_task_image_workdir keeps the last WORKDIR anywhere in the Dockerfile and claims that equals the final stage. Docker resets WORKDIR per FROM, so a builder-stage WORKDIR is used when the final stage omits one, sending agents and tests/test.sh to the wrong cwd.
Reviewed by Cursor Bugbot for commit 85343bd. Configure here.
There was a problem hiding this comment.
Checked all 89 official tasks: exactly one is multi-stage (financial-document-processor) and its final stage pins WORKDIR /app, so last-wins parses every official task correctly. Leaving the parser as is — TB2_TASK_WORKDIR covers exotic custom Dockerfiles.
| reward = float(raw) if raw else 0.0 | ||
| except ValueError: | ||
| reward = 0.0 | ||
| break |
There was a problem hiding this comment.
Timeout can drop a written reward
Medium Severity
Canonical scoring only trusts the trailing __TB2_REWARD__: echo from shell_exec output. The same timeout_s also covers staging, test.sh, tail, and that echo, so if camel truncates after test.sh has already written /logs/verifier/reward.txt, the reward is ignored and the episode scores 0.0.
Reviewed by Cursor Bugbot for commit 85343bd. Configure here.
There was a problem hiding this comment.
Working as intended: timeout_s is the task's declared verifier budget, and blowing it scores 0 in the official TB2 harness too. Recovering a reward written mid-timeout would deviate from that semantics.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f1b0b8e. Configure here.
| self._terminal_toolkit = TerminalToolkit( | ||
| timeout=self.command_timeout_s, | ||
| working_directory=str(task_dir), | ||
| working_directory=workdir, |
There was a problem hiding this comment.
Fallback pytest path left broken
Medium Severity
Removing install_dependencies=['pytest'] drops the only mechanism that ensured pytest was available for the bare-pytest fallback, while that fallback still does python -m pytest when uvx is missing. Custom task dirs without tests/test.sh on images that lack both uvx and a preinstalled pytest will fail closed with reward 0.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit f1b0b8e. Configure here.
The tbench2_env fixes are upstream now (huggingface/OpenEnv#965 + #972); embedding the installed source remains the mechanism that guarantees the sandbox runs exactly the version validated locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The server-side fixes this backend scores through (canonical test.sh evaluate, server-side WORKDIR, TB2_WITHHOLD_TESTS) merged upstream as huggingface/OpenEnv#965 + #972, so the README no longer pins the fork (whose pinned sha was orphaned by a rebase anyway); the launcher preflight still fails fast on an older install. scan_golden now takes the test-log tail from the evaluate output itself — the patched server wipes the on-disk copy with the verify window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The server-side fixes this backend scores through (canonical test.sh evaluate, server-side WORKDIR, TB2_WITHHOLD_TESTS) merged upstream as huggingface/OpenEnv#965 + #972, so the README no longer pins the fork (whose pinned sha was orphaned by a rebase anyway); the launcher preflight still fails fast on an older install. scan_golden now takes the test-log tail from the evaluate output itself — the patched server wipes the on-disk copy with the verify window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
Summary
Makes local-mode
Tbench2Environmentfaithful to real Terminal-Bench-2 semantics:evaluatescores with each task's officialtests/test.sh(pinned uvx toolchain,/logs/verifier/reward.txt), command execution honors real time budgets, and commands run in the task image's own WORKDIR. Follow-up to #503, which fixed evaluate always returning 0 — this makes the returned reward match the official TB2 verifier.Details:
pytest tests/(current behavior) skips the toolchain the official harness pins (uvx: Python 3.13, pytest 8.4.1, ctrf) and mis-scores tasks that depend on it._evaluate_tasknow runstests/test.shwhen the task ships it; the bare-pytest fallback remains for custom task dirs, upgraded to preferuvx --with pytest==8.4.1so it carries its own toolchain too.shell_exechas a per-call default of 20s and ignores the constructor timeout, silently truncating agent commands and evaluations (e.g.circuit-fibsqrtdeclares a 3600s verifier budget and was cut mid-test, scoring a false 0). Agent execs now passcommand_timeout_s(overridable viaTB2_COMMAND_TIMEOUT_S); evaluate passes the task's own[verifier].timeout_sec./app(fix-git:/app/personal-site,prove-plus-comm:/workspace). The shell cwd now resolves from the task'senvironment/Dockerfile(TB2_TASK_WORKDIRoverrides), falling back to the task source dir for plain local mode.install_dependencies=['pytest']: evaluation now carries its own pytest via uvx; when the server runs withCAMEL_RUNTIME=true, camel's.initial_envvenv then also stops shadowing the task image's native python on PATH.Type of Change
Alignment Checklist
.claude/docs/PRINCIPLES.mdand this PR aligns with our principles (notably: rewards are computed inside the environment, per RFC 002).claude/docs/INVARIANTS.mdand no invariants are violated (Gym API signatures unchanged; server-internal changes only)bash .claude/hooks/lint.sh— the touched file is clean (pre-existing failures in other envs are untouched)RFC Status
Test Plan
Validated on real per-task TB2 containers (official task images), scoring official solutions under the official oracle convention:
setuptoolsfor pystan, etc. — happy to share the per-task evidence).circuit-fibsqrt/compile-compcert(timeouts),fix-git/prove-plus-comm(workdir), plus correct scoring across the suite via test.sh.Note: the canonical eval uses the official global paths (
/tests,/logs/verifier/reward.txt), which assumes one concurrent session per container — consistent with TB2's one-task-per-container model (and with one-env-one-trajectory); flagging for reviewers running multi-session servers against a single container.Claude Code Review
/alignment-review run on this change: Tier 1 (format) fixed; flags addressed during review — fallback now carries its own pytest toolchain; workdir resolution centralized server-side so all deployment modes agree. No principle or RFC conflicts identified.
Note
Medium Risk
Changes reward computation and shell cwd/timeouts for all local TB2 sessions; canonical eval uses shared
/testspaths and requires the lock when multiple sessions share one container.Overview
Aligns local-mode
Tbench2Environmentwith official Terminal-Bench 2 verifier behavior so rewards and execution context match what agents and the harness expect.Evaluation now prefers each task’s
tests/test.shwhen present: tests are staged under/tests, the harness runs from the resolved task workdir, and reward is read from/logs/verifier/reward.txtinstead of inferring pass/fail from barepytest. A thread lock serializes canonical evals so concurrent sessions on one server do not clash on/testsor the reward file. Tasks withouttest.shkeep a pytest fallback, now preferringuvx --with pytest==8.4.1and using the agent workdir; verifier runs use[verifier].timeout_secfromtask.toml.Shell execution passes
timeout=self.command_timeout_sintoshell_exec(camel’s per-call 20s default was truncating long commands).command_timeout_sdefaults viaTB2_COMMAND_TIMEOUT_Swhen the env is constructed with no args.Working directory is resolved on
resetfromTB2_TASK_WORKDIR, else the taskenvironment/DockerfileWORKDIR(ignored when it overlaps the server install tree), else the task source dir—replacing always using the task folder as cwd.install_dependencies=['pytest']is removed fromTerminalToolkitbecause scoring brings its own toolchain.Tests add coverage for Dockerfile workdir selection and server-tree rejection.
Reviewed by Cursor Bugbot for commit f1b0b8e. Bugbot is set up for automated code reviews on this repo. Configure here.