Fix public task BenchFlow run path - #23
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dabad2be24
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| DOCS="${DOCS_URL:-http://localhost:9004}" | ||
| DRIVE="${DRIVE_URL:-http://localhost:9005}" | ||
| DRIVE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}" |
There was a problem hiding this comment.
Use canonical MOCK_ Drive URL overrides*
The repo contract in AGENTS.md says to prefer current MOCK_* service names and not add legacy service-name contracts. This new GDRIVE_URL/DRIVE_URL chain establishes another public Drive URL override and still ignores the canonical MOCK_GDRIVE_URL exported by docker/Dockerfile.base, so non-local runs that override the canonical service URL will fall back to localhost; please key these touched task scripts off MOCK_GDRIVE_URL first instead of adding GDRIVE_URL.
Useful? React with 👍 / 👎.
Summary
tasks/_manifests/env-0.tomlValidation
docker/build-base.shfor f in tasks/*/oracle/solve.sh tasks/*/verifier/test.sh; do bash -n "$f" || exit 1; donegit diff --checkuvx --from benchflow==0.6.4 bench tasks check <six coverage tasks> --level structuraluvx --from benchflow==0.6.4 bench tasks check <six coverage tasks> --level runtime-capability --sandbox dockerBENCHFLOW_REWARD_LENIENT=1 uvx --from benchflow==0.6.4 bench eval run ... --agent oracle --sandbox docker --include <all 13 public tasks>-> 13/13, errors=0, idle_timeouts=0codex exec -m gpt-5.5 "Reply exactly ok"using temporaryCODEX_HOMEcopied from local subscription auth -> okBENCHFLOW_REWARD_LENIENT=1 uvx --from benchflow==0.6.4 bench eval run ... --agent codex-acp --model gpt-5.5 --include <six env coverage tasks>-> 6/6 environments started/scored, 3/6 passed, errors=0, idle_timeouts=0, one task hit task wall-clock timeoutClaude Code status
claude -p --model opus --effort max --max-budget-usd 1 "Reply with exactly: ok"-> okclaude-agent-acplocal-login probe without transferable credentials -> blocked before task withANTHROPIC_API_KEY required...claude-agent-acpwith existingCLAUDE_CODE_OAUTH_TOKEN-> sandbox starts but fails with401 Invalid bearer tokenclaude-agent-acpwithANTHROPIC_API_KEYon this machine still returned ACPAuthentication required; stopped after repeated auth failures on the first taskThe public mock environments are now oracle-runnable end to end; Claude Code subscription execution still requires a fresh transferable Claude credential (
~/.claude/.credentials.jsonor valid OAuth token), documented in the new guide.