Skip to content

Fix public task BenchFlow run path - #23

Merged
bingran-you merged 1 commit into
mainfrom
bry/public-benchflow-e2e-guide-and-task-fixes
Jul 4, 2026
Merged

Fix public task BenchFlow run path#23
bingran-you merged 1 commit into
mainfrom
bry/public-benchflow-e2e-guide-and-task-fixes

Conversation

@bingran-you

Copy link
Copy Markdown
Contributor

Summary

  • align public task oracle/verifier service defaults with tasks/_manifests/env-0.toml
  • fix the Drive permissions oracle to find target files by exact name instead of relying on one broad Drive list page
  • add a self-contained public guide for running tasks with BenchFlow, Codex subscription auth, and Claude Code preflight/status

Validation

  • docker/build-base.sh
  • for f in tasks/*/oracle/solve.sh tasks/*/verifier/test.sh; do bash -n "$f" || exit 1; done
  • git diff --check
  • uvx --from benchflow==0.6.4 bench tasks check <six coverage tasks> --level structural
  • uvx --from benchflow==0.6.4 bench tasks check <six coverage tasks> --level runtime-capability --sandbox docker
  • BENCHFLOW_REWARD_LENIENT=1 uvx --from benchflow==0.6.4 bench eval run ... --agent oracle --sandbox docker --include <all 13 public tasks> -> 13/13, errors=0, idle_timeouts=0
  • codex exec -m gpt-5.5 "Reply exactly ok" using temporary CODEX_HOME copied from local subscription auth -> ok
  • BENCHFLOW_REWARD_LENIENT=1 uvx --from benchflow==0.6.4 bench eval run ... --agent codex-acp --model gpt-5.5 --include <six env coverage tasks> -> 6/6 environments started/scored, 3/6 passed, errors=0, idle_timeouts=0, one task hit task wall-clock timeout

Claude Code status

  • host claude -p --model opus --effort max --max-budget-usd 1 "Reply with exactly: ok" -> ok
  • BenchFlow claude-agent-acp local-login probe without transferable credentials -> blocked before task with ANTHROPIC_API_KEY required...
  • BenchFlow claude-agent-acp with existing CLAUDE_CODE_OAUTH_TOKEN -> sandbox starts but fails with 401 Invalid bearer token
  • BenchFlow claude-agent-acp with ANTHROPIC_API_KEY on this machine still returned ACP Authentication required; stopped after repeated auth failures on the first task

The public mock environments are now oracle-runnable end to end; Claude Code subscription execution still requires a fresh transferable Claude credential (~/.claude/.credentials.json or valid OAuth token), documented in the new guide.

@bingran-you
bingran-you merged commit 9e401fd into main Jul 4, 2026
11 checks passed
@bingran-you
bingran-you deleted the bry/public-benchflow-e2e-guide-and-task-fixes branch July 4, 2026 23:32

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dabad2be24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


DOCS="${DOCS_URL:-http://localhost:9004}"
DRIVE="${DRIVE_URL:-http://localhost:9005}"
DRIVE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use canonical MOCK_ Drive URL overrides*

The repo contract in AGENTS.md says to prefer current MOCK_* service names and not add legacy service-name contracts. This new GDRIVE_URL/DRIVE_URL chain establishes another public Drive URL override and still ignores the canonical MOCK_GDRIVE_URL exported by docker/Dockerfile.base, so non-local runs that override the canonical service URL will fall back to localhost; please key these touched task scripts off MOCK_GDRIVE_URL first instead of adding GDRIVE_URL.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant