From dabad2be241ec600e58f0446d2d84f04a4a11d2b Mon Sep 17 00:00:00 2001 From: Bingran You Date: Sat, 4 Jul 2026 16:27:30 -0700 Subject: [PATCH] Fix public task BenchFlow run path --- README.md | 1 + docs/README.md | 2 + docs/guides/run-tasks-with-benchflow.md | 240 ++++++++++++++++++ docs/validated-workflows.md | 31 +++ .../verifier/test.sh | 2 +- .../verifier/test.sh | 2 +- .../oracle/solve.sh | 29 ++- .../verifier/test.sh | 2 +- .../oracle/solve.sh | 8 +- .../verifier/test.sh | 6 +- tasks/multi-mail-cal-sync/verifier/test.sh | 2 +- tasks/slack-channel-reorg/oracle/solve.sh | 2 +- tasks/slack-channel-reorg/verifier/test.sh | 2 +- .../oracle/solve.sh | 2 +- .../verifier/test.sh | 2 +- 15 files changed, 307 insertions(+), 26 deletions(-) create mode 100644 docs/guides/run-tasks-with-benchflow.md diff --git a/README.md b/README.md index ffc859463..36752ef69 100644 --- a/README.md +++ b/README.md @@ -196,6 +196,7 @@ env0/ - [API validation playbook](docs/api-validation-playbook.md) - [Parity audit](docs/parity-audit/README.md) - [Validated workflows](docs/validated-workflows.md) +- [Run public tasks with BenchFlow](docs/guides/run-tasks-with-benchflow.md) - [Good first contributions](docs/good-first-contributions.md) - [Contributing](CONTRIBUTING.md) - [Security policy](SECURITY.md) diff --git a/docs/README.md b/docs/README.md index 3128d8650..25740f811 100644 --- a/docs/README.md +++ b/docs/README.md @@ -19,6 +19,8 @@ validation. conformance status. - [Validated workflows](validated-workflows.md) — commands that have been run against this checkout and the intended preconditions for heavier commands. +- [Run public tasks with BenchFlow](guides/run-tasks-with-benchflow.md) — + self-contained setup and subscription-agent task runs for public tasks. - [Contributing](../CONTRIBUTING.md) — repo boundaries, validation matrix, and pull request expectations. - [Security policy](../SECURITY.md) — private vulnerability reporting and diff --git a/docs/guides/run-tasks-with-benchflow.md b/docs/guides/run-tasks-with-benchflow.md new file mode 100644 index 000000000..fc4d8e7e7 --- /dev/null +++ b/docs/guides/run-tasks-with-benchflow.md @@ -0,0 +1,240 @@ +# Run Public Tasks With BenchFlow + +This guide is the self-contained path for a new contributor to clone `env0`, +build the public mock environment image, and run real task packages end to end +with BenchFlow. + +Validated on 2026-07-04 with: + +- `benchflow==0.6.4` +- Docker Desktop 29.3.0 +- Codex CLI 0.142.4 using local subscription auth +- Claude Code 2.1.185 host login preflight + +The public task runtime uses `tasks/_manifests/env-0.toml` and these service +ports: auth `9000`, Gmail `9001`, Calendar `9002`, Drive `9003`, Docs `9004`, +Slack `9005`, Discord `9006`, and Stripe `9007`. + +## 1. Fresh Checkout + +Install Git, Docker, Python 3.12+, and `uv`. Then clone this repository: + +```bash +git clone https://github.com/benchflow-ai/env0.git +cd env0 +``` + +Use the pinned BenchFlow version validated by this guide: + +```bash +export BENCHFLOW_VERSION=0.6.4 +uvx --from "benchflow==${BENCHFLOW_VERSION}" bench --version +``` + +Expected output: + +```text +benchflow 0.6.4 +``` + +On Apple Silicon, build and run the Linux/amd64 task images: + +```bash +export DOCKER_DEFAULT_PLATFORM=linux/amd64 +``` + +## 2. Build The Public Base Image + +```bash +docker/build-base.sh +``` + +This builds: + +```text +ghcr.io/benchflow-ai/env0:0.1.0 +ghcr.io/benchflow-ai/env0:latest +``` + +You can also run from a published base image after maintainers push it, but a +fresh contributor should be able to build locally with the command above. + +## 3. Check Task Packages + +This six-task matrix covers every public mock service at least once: + +| Task | Services covered | +|---|---| +| `auth-least-privilege-summary` | `mock-auth`, `mock-gmail` | +| `discord-incident-followup` | `mock-discord` | +| `gcal-federal-register-meeting-amendments` | `mock-gcal` | +| `gdoc-search-keyword-index` | `mock-gdrive`, `mock-gdoc` | +| `slack-search-channel-history` | `mock-slack` | +| `stripe-refund-correct-customer` | `mock-stripe` | + +Run structural and runtime-capability checks: + +```bash +for task in \ + auth-least-privilege-summary \ + discord-incident-followup \ + gcal-federal-register-meeting-amendments \ + gdoc-search-keyword-index \ + slack-search-channel-history \ + stripe-refund-correct-customer +do + uvx --from "benchflow==${BENCHFLOW_VERSION}" bench tasks check \ + "tasks/${task}" --level structural + uvx --from "benchflow==${BENCHFLOW_VERSION}" bench tasks check \ + "tasks/${task}" --level runtime-capability --sandbox docker +done +``` + +## 4. Run The Oracle Baseline + +Run all public task packages with shipped oracle solutions: + +```bash +export BENCHFLOW_REWARD_LENIENT=1 + +uvx --from "benchflow==${BENCHFLOW_VERSION}" bench eval run \ + --tasks-dir tasks \ + --include auth-least-privilege-summary \ + --include discord-incident-followup \ + --include email-confidential-forward \ + --include email-no-wrong-recipients \ + --include email-vendor-report-organize \ + --include gcal-federal-register-meeting-amendments \ + --include gdoc-search-keyword-index \ + --include gdrive-sensitive-file-lockdown \ + --include multi-doc-slack-spec-drift \ + --include multi-mail-cal-sync \ + --include slack-channel-reorg \ + --include slack-search-channel-history \ + --include stripe-refund-correct-customer \ + --agent oracle \ + --sandbox docker \ + --context-root . \ + --concurrency 1 \ + --build-concurrency 1 \ + --jobs-dir .local/bf-oracle-all-public +``` + +Expected result for this revision: + +```text +Job complete: 13/13 (100.0%), errors=0, idle_timeouts=0 +``` + +## 5. Run With Codex Subscription Auth + +Log in locally first: + +```bash +codex login +test -s "$HOME/.codex/auth.json" +``` + +Probe the local subscription auth: + +```bash +CODEX_HOME="$(mktemp -d /tmp/codex-home.XXXXXX)" +mkdir -p "$CODEX_HOME" +cp "$HOME/.codex/auth.json" "$CODEX_HOME/auth.json" +printf 'model = "gpt-5.5"\nmodel_reasoning_effort = "xhigh"\n' > "$CODEX_HOME/config.toml" +env -u OPENAI_API_KEY -u OPENAI_BASE_URL CODEX_HOME="$CODEX_HOME" \ + codex exec -m gpt-5.5 "Reply exactly ok" +``` + +Run the six-task environment coverage matrix: + +```bash +unset OPENAI_API_KEY OPENAI_BASE_URL +export CODEX_AUTH_JSON="$(tr -d '\n' < "$HOME/.codex/auth.json")" +export CODEX_CONFIG='{"model":"gpt-5.5","model_reasoning_effort":"xhigh"}' + +uvx --from "benchflow==${BENCHFLOW_VERSION}" bench eval run \ + --tasks-dir tasks \ + --include auth-least-privilege-summary \ + --include discord-incident-followup \ + --include gcal-federal-register-meeting-amendments \ + --include gdoc-search-keyword-index \ + --include slack-search-channel-history \ + --include stripe-refund-correct-customer \ + --agent codex-acp \ + --model gpt-5.5 \ + --agent-env CODEX_AUTH_JSON="${CODEX_AUTH_JSON}" \ + --agent-env CODEX_CONFIG="${CODEX_CONFIG}" \ + --sandbox docker \ + --context-root . \ + --concurrency 1 \ + --build-concurrency 1 \ + --agent-idle-timeout 900 \ + --jobs-dir .local/bf-codex-gpt55-env-coverage +``` + +Do not pass `--reasoning-effort` to `codex-acp` on BenchFlow 0.6.4. Put Codex +reasoning settings in `CODEX_CONFIG` as shown above. + +The validated run for this revision started and scored all six environments +with `errors=0` and `idle_timeouts=0`. Model pass rate is not the same as +environment health: the validated Codex run passed 3/6 tasks, failed 2/6 by +verifier score, and hit the task wall-clock timeout on 1/6. + +## 6. Claude Code Status + +Host Claude Code subscription login can be checked with: + +```bash +claude -p --model opus --effort max --max-budget-usd 1 "Reply with exactly: ok" +``` + +BenchFlow 0.6.4's `claude-agent-acp` path needs transferable credentials inside +the Docker sandbox. A host login that works through the local keychain is not +enough unless BenchFlow can see one of these: + +- `~/.claude/.credentials.json` +- a valid `CLAUDE_CODE_OAUTH_TOKEN` +- a valid `CLAUDE_OAUTH_TOKEN` + +Before running a Claude task, check for one of those credentials: + +```bash +test -f "$HOME/.claude/.credentials.json" || \ + test -n "${CLAUDE_CODE_OAUTH_TOKEN:-}" || \ + test -n "${CLAUDE_OAUTH_TOKEN:-}" +``` + +Then run a one-task probe: + +```bash +uvx --from "benchflow==${BENCHFLOW_VERSION}" bench eval run \ + --tasks-dir tasks \ + --include discord-incident-followup \ + --agent claude-agent-acp \ + --model opus \ + --sandbox docker \ + --context-root . \ + --concurrency 1 \ + --build-concurrency 1 \ + --agent-idle-timeout 900 \ + --jobs-dir .local/bf-claude-opus-probe +``` + +If BenchFlow fails before starting Docker with `ANTHROPIC_API_KEY required`, it +did not find transferable Claude credentials. If the agent starts and then fails +with `401 Invalid bearer token`, the exported OAuth token is stale. Refresh the +Claude Code login/token locally before rerunning. + +Do not pass `--reasoning-effort` to `claude-agent-acp` on BenchFlow 0.6.4. + +## Troubleshooting + +- If Docker on Apple Silicon builds or runs the wrong architecture, re-export + `DOCKER_DEFAULT_PLATFORM=linux/amd64`. +- If a verifier reaches the wrong service, compare the script default with + `tasks/_manifests/env-0.toml`. +- If Codex starts but uses API-key auth instead of subscription auth, unset + `OPENAI_API_KEY` and `OPENAI_BASE_URL`, then pass `CODEX_AUTH_JSON`. +- If a model task times out, inspect the task's `timeout_sec` in `task.md` and + the per-task `result.json` under the chosen `--jobs-dir`. diff --git a/docs/validated-workflows.md b/docs/validated-workflows.md index 0aa75c35f..1c38c4e0a 100644 --- a/docs/validated-workflows.md +++ b/docs/validated-workflows.md @@ -32,6 +32,37 @@ The `bench eval run` command is the end-to-end task validation path. It verifies that BenchFlow can build public task images, start the manifest-declared `mock-*` services, run oracle solutions, and score verifiers. +For the public `tasks/` reference set and local Codex/Claude agent setup, use +the self-contained guide in +[`docs/guides/run-tasks-with-benchflow.md`](guides/run-tasks-with-benchflow.md). +The all-task oracle baseline for the current public task set is: + +```bash +BENCHFLOW_REWARD_LENIENT=1 bench eval run \ + --tasks-dir tasks \ + --include auth-least-privilege-summary \ + --include discord-incident-followup \ + --include email-confidential-forward \ + --include email-no-wrong-recipients \ + --include email-vendor-report-organize \ + --include gcal-federal-register-meeting-amendments \ + --include gdoc-search-keyword-index \ + --include gdrive-sensitive-file-lockdown \ + --include multi-doc-slack-spec-drift \ + --include multi-mail-cal-sync \ + --include slack-channel-reorg \ + --include slack-search-channel-history \ + --include stripe-refund-correct-customer \ + --agent oracle \ + --sandbox docker \ + --context-root . \ + --concurrency 1 \ + --build-concurrency 1 \ + --jobs-dir .local/bf-oracle-all-public +``` + +That command should complete with `13/13`, `errors=0`, and `idle_timeouts=0`. + Maintainers can publish the release image with the `Publish Base Image` GitHub Actions workflow. The workflow uses the repository `GITHUB_TOKEN` with `packages: write`, pushes `ghcr.io/benchflow-ai/env0:` and `latest`, diff --git a/tasks/gcal-federal-register-meeting-amendments/verifier/test.sh b/tasks/gcal-federal-register-meeting-amendments/verifier/test.sh index 9a78d8b67..422624a6c 100755 --- a/tasks/gcal-federal-register-meeting-amendments/verifier/test.sh +++ b/tasks/gcal-federal-register-meeting-amendments/verifier/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -euo pipefail -GCAL="${CALENDAR_URL:-http://localhost:9003}" +GCAL="${GCAL_URL:-${CALENDAR_URL:-http://localhost:9002}}" LOGS_DIR="${LOGS_DIR:-/logs/verifier}" mkdir -p "$LOGS_DIR" diff --git a/tasks/gdoc-search-keyword-index/verifier/test.sh b/tasks/gdoc-search-keyword-index/verifier/test.sh index ff5944fa7..87ec02507 100755 --- a/tasks/gdoc-search-keyword-index/verifier/test.sh +++ b/tasks/gdoc-search-keyword-index/verifier/test.sh @@ -3,7 +3,7 @@ set -euo pipefail DOCS="${DOCS_URL:-http://localhost:9004}" -DRIVE="${DRIVE_URL:-http://localhost:9005}" +DRIVE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}" LOGS_DIR="${LOGS_DIR:-/logs/verifier}" mkdir -p "$LOGS_DIR" diff --git a/tasks/gdrive-sensitive-file-lockdown/oracle/solve.sh b/tasks/gdrive-sensitive-file-lockdown/oracle/solve.sh index 93314749f..c97e9d255 100755 --- a/tasks/gdrive-sensitive-file-lockdown/oracle/solve.sh +++ b/tasks/gdrive-sensitive-file-lockdown/oracle/solve.sh @@ -4,9 +4,6 @@ # public blog drafts) from problematic overshares on sensitive files. set -euo pipefail -# Step 1: List all files -files_json=$(gws drive files list --params '{"fields": "files(id,name,mimeType)", "pageSize": 100}') - # Sensitive file names to lock down SENSITIVE_NAMES=( "api-keys.env" @@ -16,15 +13,25 @@ SENSITIVE_NAMES=( ) for name in "${SENSITIVE_NAMES[@]}"; do - # Find the file ID - file_id=$(echo "$files_json" | python3 -c " + # Find the file ID by exact name. The task image contains more than one page + # of Drive items, so a single broad files.list can miss target files. + file_id=$(gws drive files list \ + --params "$(python3 - "$name" <<'PY' +import json +import sys + +name = sys.argv[1] +escaped = name.replace("'", "\\'") +print(json.dumps({ + "q": f"name = '{escaped}'", + "fields": "files(id,name,mimeType)", + "pageSize": 10, +})) +PY +)" | python3 -c " import sys, json -name = '''${name}''' -files = json.load(sys.stdin)['files'] -for f in files: - if f['name'] == name: - print(f['id']) - break +files = json.load(sys.stdin).get('files', []) +print(files[0]['id'] if files else '') ") if [ -z "$file_id" ]; then diff --git a/tasks/gdrive-sensitive-file-lockdown/verifier/test.sh b/tasks/gdrive-sensitive-file-lockdown/verifier/test.sh index 446fa2da2..1a7b3c673 100755 --- a/tasks/gdrive-sensitive-file-lockdown/verifier/test.sh +++ b/tasks/gdrive-sensitive-file-lockdown/verifier/test.sh @@ -2,7 +2,7 @@ # Fetch state/diff/action_log from the environment and run evaluate.py set -euo pipefail -BASE="${DRIVE_URL:-http://localhost:9005}" +BASE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}" LOGS_DIR="${LOGS_DIR:-/logs/verifier}" mkdir -p "$LOGS_DIR" diff --git a/tasks/multi-doc-slack-spec-drift/oracle/solve.sh b/tasks/multi-doc-slack-spec-drift/oracle/solve.sh index d5525a259..eab548fc0 100755 --- a/tasks/multi-doc-slack-spec-drift/oracle/solve.sh +++ b/tasks/multi-doc-slack-spec-drift/oracle/solve.sh @@ -8,8 +8,8 @@ set -euo pipefail # 4. Add comments for each drift DOCS="${DOCS_URL:-http://localhost:9004}" -DRIVE="${DRIVE_URL:-http://localhost:9005}" -SLACK="${SLACK_URL:-http://localhost:9002}" +DRIVE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}" +SLACK="${SLACK_URL:-http://localhost:9005}" python3 << 'PYEOF' import json @@ -19,8 +19,8 @@ import sys import urllib.request DOCS = os.environ.get("DOCS_URL", "http://localhost:9004") -DRIVE = os.environ.get("DRIVE_URL", "http://localhost:9005") -SLACK = os.environ.get("SLACK_URL", "http://localhost:9002") +DRIVE = os.environ.get("GDRIVE_URL") or os.environ.get("DRIVE_URL", "http://localhost:9003") +SLACK = os.environ.get("SLACK_URL", "http://localhost:9005") def gws(*args): """Run a gws command and return parsed JSON.""" diff --git a/tasks/multi-doc-slack-spec-drift/verifier/test.sh b/tasks/multi-doc-slack-spec-drift/verifier/test.sh index 41264c7ef..0bd801609 100755 --- a/tasks/multi-doc-slack-spec-drift/verifier/test.sh +++ b/tasks/multi-doc-slack-spec-drift/verifier/test.sh @@ -2,8 +2,8 @@ set -euo pipefail DOCS="${DOCS_URL:-http://localhost:9004}" -DRIVE="${DRIVE_URL:-http://localhost:9005}" -SLACK="${SLACK_URL:-http://localhost:9002}" +DRIVE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}" +SLACK="${SLACK_URL:-http://localhost:9005}" LOGS_DIR="${LOGS_DIR:-/logs/verifier}" DOC_ID="1ApiRateLimitingPolicySpec000000000000000000" mkdir -p "$LOGS_DIR" @@ -12,7 +12,7 @@ curl -s "$DOCS/_admin/state" > /tmp/final_state.json curl -s "$DOCS/_admin/diff" > /tmp/docs_diff.json # Agents may create comments via the Docs API (port 9004) or the -# Drive comments API (port 9005). Fetch from both and merge. +# Drive comments API (port 9003). Fetch from both and merge. curl -s "$DRIVE/drive/v3/files/${DOC_ID}/comments" > /tmp/drive_comments.json 2>/dev/null || echo '{"comments":[]}' > /tmp/drive_comments.json python3 -c " diff --git a/tasks/multi-mail-cal-sync/verifier/test.sh b/tasks/multi-mail-cal-sync/verifier/test.sh index 69266d815..7cbeee149 100755 --- a/tasks/multi-mail-cal-sync/verifier/test.sh +++ b/tasks/multi-mail-cal-sync/verifier/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -euo pipefail -GCAL="${CALENDAR_URL:-http://localhost:9003}" +GCAL="${GCAL_URL:-${CALENDAR_URL:-http://localhost:9002}}" GMAIL="${GMAIL_URL:-http://localhost:9001}" LOGS_DIR="${LOGS_DIR:-/logs/verifier}" mkdir -p "$LOGS_DIR" diff --git a/tasks/slack-channel-reorg/oracle/solve.sh b/tasks/slack-channel-reorg/oracle/solve.sh index 079ea8338..3ff753bdf 100755 --- a/tasks/slack-channel-reorg/oracle/solve.sh +++ b/tasks/slack-channel-reorg/oracle/solve.sh @@ -15,7 +15,7 @@ set -euo pipefail -BASE="${SLACK_URL:-http://localhost:9002}" +BASE="${SLACK_URL:-http://localhost:9005}" BOT="Authorization: Bearer ${SLACK_BOT_TOKEN:-mock-bot-token}" WS="X-Mock-Slack-Workspace: workspace_001" diff --git a/tasks/slack-channel-reorg/verifier/test.sh b/tasks/slack-channel-reorg/verifier/test.sh index 66ab17dc3..7ceea7bf7 100755 --- a/tasks/slack-channel-reorg/verifier/test.sh +++ b/tasks/slack-channel-reorg/verifier/test.sh @@ -1,7 +1,7 @@ #!/usr/bin/env bash set -euo pipefail -BASE="${SLACK_URL:-http://localhost:9002}" +BASE="${SLACK_URL:-http://localhost:9005}" LOGS_DIR="${LOGS_DIR:-/logs/verifier}" mkdir -p "$LOGS_DIR" diff --git a/tasks/slack-search-channel-history/oracle/solve.sh b/tasks/slack-search-channel-history/oracle/solve.sh index 5cf9853d9..c9a9a729d 100755 --- a/tasks/slack-search-channel-history/oracle/solve.sh +++ b/tasks/slack-search-channel-history/oracle/solve.sh @@ -15,7 +15,7 @@ set -euo pipefail -BASE="${SLACK_URL:-http://localhost:9002}" +BASE="${SLACK_URL:-http://localhost:9005}" BOT="Authorization: Bearer ${SLACK_BOT_TOKEN:-mock-bot-token}" USER_TOK="Authorization: Bearer ${SLACK_USER_TOKEN:-mock-user-token}" diff --git a/tasks/slack-search-channel-history/verifier/test.sh b/tasks/slack-search-channel-history/verifier/test.sh index 926e9cfc7..0cc25c40b 100755 --- a/tasks/slack-search-channel-history/verifier/test.sh +++ b/tasks/slack-search-channel-history/verifier/test.sh @@ -2,7 +2,7 @@ # Fetch state/diff/action_log from slack and run evaluate.py set -euo pipefail -BASE="${SLACK_URL:-http://localhost:9002}" +BASE="${SLACK_URL:-http://localhost:9005}" LOGS_DIR="${LOGS_DIR:-/logs/verifier}" mkdir -p "$LOGS_DIR"