Fix public task BenchFlow run path

README.md

@@ -196,6 +196,7 @@ env0/
 - [API validation playbook](docs/api-validation-playbook.md)
 - [Parity audit](docs/parity-audit/README.md)
 - [Validated workflows](docs/validated-workflows.md)
+- [Run public tasks with BenchFlow](docs/guides/run-tasks-with-benchflow.md)
 - [Good first contributions](docs/good-first-contributions.md)
 - [Contributing](CONTRIBUTING.md)
 - [Security policy](SECURITY.md)

docs/README.md

@@ -19,6 +19,8 @@ validation.
   conformance status.
 - [Validated workflows](validated-workflows.md) — commands that have been run
   against this checkout and the intended preconditions for heavier commands.
+- [Run public tasks with BenchFlow](guides/run-tasks-with-benchflow.md) —
+  self-contained setup and subscription-agent task runs for public tasks.
 - [Contributing](../CONTRIBUTING.md) — repo boundaries, validation matrix, and
   pull request expectations.
 - [Security policy](../SECURITY.md) — private vulnerability reporting and

docs/guides/run-tasks-with-benchflow.md

@@ -0,0 +1,240 @@
+# Run Public Tasks With BenchFlow
+
+This guide is the self-contained path for a new contributor to clone `env0`,
+build the public mock environment image, and run real task packages end to end
+with BenchFlow.
+
+Validated on 2026-07-04 with:
+
+- `benchflow==0.6.4`
+- Docker Desktop 29.3.0
+- Codex CLI 0.142.4 using local subscription auth
+- Claude Code 2.1.185 host login preflight
+
+The public task runtime uses `tasks/_manifests/env-0.toml` and these service
+ports: auth `9000`, Gmail `9001`, Calendar `9002`, Drive `9003`, Docs `9004`,
+Slack `9005`, Discord `9006`, and Stripe `9007`.
+
+## 1. Fresh Checkout
+
+Install Git, Docker, Python 3.12+, and `uv`. Then clone this repository:
+
+```bash
+git clone https://github.com/benchflow-ai/env0.git
+cd env0
+```
+
+Use the pinned BenchFlow version validated by this guide:
+
+```bash
+export BENCHFLOW_VERSION=0.6.4
+uvx --from "benchflow==${BENCHFLOW_VERSION}" bench --version
+```
+
+Expected output:
+
+```text
+benchflow 0.6.4
+```
+
+On Apple Silicon, build and run the Linux/amd64 task images:
+
+```bash
+export DOCKER_DEFAULT_PLATFORM=linux/amd64
+```
+
+## 2. Build The Public Base Image
+
+```bash
+docker/build-base.sh
+```
+
+This builds:
+
+```text
+ghcr.io/benchflow-ai/env0:0.1.0
+ghcr.io/benchflow-ai/env0:latest
+```
+
+You can also run from a published base image after maintainers push it, but a
+fresh contributor should be able to build locally with the command above.
+
+## 3. Check Task Packages
+
+This six-task matrix covers every public mock service at least once:
+
+| Task | Services covered |
+|---|---|
+| `auth-least-privilege-summary` | `mock-auth`, `mock-gmail` |
+| `discord-incident-followup` | `mock-discord` |
+| `gcal-federal-register-meeting-amendments` | `mock-gcal` |
+| `gdoc-search-keyword-index` | `mock-gdrive`, `mock-gdoc` |
+| `slack-search-channel-history` | `mock-slack` |
+| `stripe-refund-correct-customer` | `mock-stripe` |
+
+Run structural and runtime-capability checks:
+
+```bash
+for task in \
+  auth-least-privilege-summary \
+  discord-incident-followup \
+  gcal-federal-register-meeting-amendments \
+  gdoc-search-keyword-index \
+  slack-search-channel-history \
+  stripe-refund-correct-customer
+do
+  uvx --from "benchflow==${BENCHFLOW_VERSION}" bench tasks check \
+    "tasks/${task}" --level structural
+  uvx --from "benchflow==${BENCHFLOW_VERSION}" bench tasks check \
+    "tasks/${task}" --level runtime-capability --sandbox docker
+done
+```
+
+## 4. Run The Oracle Baseline
+
+Run all public task packages with shipped oracle solutions:
+
+```bash
+export BENCHFLOW_REWARD_LENIENT=1
+
+uvx --from "benchflow==${BENCHFLOW_VERSION}" bench eval run \
+  --tasks-dir tasks \
+  --include auth-least-privilege-summary \
+  --include discord-incident-followup \
+  --include email-confidential-forward \
+  --include email-no-wrong-recipients \
+  --include email-vendor-report-organize \
+  --include gcal-federal-register-meeting-amendments \
+  --include gdoc-search-keyword-index \
+  --include gdrive-sensitive-file-lockdown \
+  --include multi-doc-slack-spec-drift \
+  --include multi-mail-cal-sync \
+  --include slack-channel-reorg \
+  --include slack-search-channel-history \
+  --include stripe-refund-correct-customer \
+  --agent oracle \
+  --sandbox docker \
+  --context-root . \
+  --concurrency 1 \
+  --build-concurrency 1 \
+  --jobs-dir .local/bf-oracle-all-public
+```
+
+Expected result for this revision:
+
+```text
+Job complete: 13/13 (100.0%), errors=0, idle_timeouts=0
+```
+
+## 5. Run With Codex Subscription Auth
+
+Log in locally first:
+
+```bash
+codex login
+test -s "$HOME/.codex/auth.json"
+```
+
+Probe the local subscription auth:
+
+```bash
+CODEX_HOME="$(mktemp -d /tmp/codex-home.XXXXXX)"
+mkdir -p "$CODEX_HOME"
+cp "$HOME/.codex/auth.json" "$CODEX_HOME/auth.json"
+printf 'model = "gpt-5.5"\nmodel_reasoning_effort = "xhigh"\n' > "$CODEX_HOME/config.toml"
+env -u OPENAI_API_KEY -u OPENAI_BASE_URL CODEX_HOME="$CODEX_HOME" \
+  codex exec -m gpt-5.5 "Reply exactly ok"
+```
+
+Run the six-task environment coverage matrix:
+
+```bash
+unset OPENAI_API_KEY OPENAI_BASE_URL
+export CODEX_AUTH_JSON="$(tr -d '\n' < "$HOME/.codex/auth.json")"
+export CODEX_CONFIG='{"model":"gpt-5.5","model_reasoning_effort":"xhigh"}'
+
+uvx --from "benchflow==${BENCHFLOW_VERSION}" bench eval run \
+  --tasks-dir tasks \
+  --include auth-least-privilege-summary \
+  --include discord-incident-followup \
+  --include gcal-federal-register-meeting-amendments \
+  --include gdoc-search-keyword-index \
+  --include slack-search-channel-history \
+  --include stripe-refund-correct-customer \
+  --agent codex-acp \
+  --model gpt-5.5 \
+  --agent-env CODEX_AUTH_JSON="${CODEX_AUTH_JSON}" \
+  --agent-env CODEX_CONFIG="${CODEX_CONFIG}" \
+  --sandbox docker \
+  --context-root . \
+  --concurrency 1 \
+  --build-concurrency 1 \
+  --agent-idle-timeout 900 \
+  --jobs-dir .local/bf-codex-gpt55-env-coverage
+```
+
+Do not pass `--reasoning-effort` to `codex-acp` on BenchFlow 0.6.4. Put Codex
+reasoning settings in `CODEX_CONFIG` as shown above.
+
+The validated run for this revision started and scored all six environments
+with `errors=0` and `idle_timeouts=0`. Model pass rate is not the same as
+environment health: the validated Codex run passed 3/6 tasks, failed 2/6 by
+verifier score, and hit the task wall-clock timeout on 1/6.
+
+## 6. Claude Code Status
+
+Host Claude Code subscription login can be checked with:
+
+```bash
+claude -p --model opus --effort max --max-budget-usd 1 "Reply with exactly: ok"
+```
+
+BenchFlow 0.6.4's `claude-agent-acp` path needs transferable credentials inside
+the Docker sandbox. A host login that works through the local keychain is not
+enough unless BenchFlow can see one of these:
+
+- `~/.claude/.credentials.json`
+- a valid `CLAUDE_CODE_OAUTH_TOKEN`
+- a valid `CLAUDE_OAUTH_TOKEN`
+
+Before running a Claude task, check for one of those credentials:
+
+```bash
+test -f "$HOME/.claude/.credentials.json" || \
+  test -n "${CLAUDE_CODE_OAUTH_TOKEN:-}" || \
+  test -n "${CLAUDE_OAUTH_TOKEN:-}"
+```
+
+Then run a one-task probe:
+
+```bash
+uvx --from "benchflow==${BENCHFLOW_VERSION}" bench eval run \
+  --tasks-dir tasks \
+  --include discord-incident-followup \
+  --agent claude-agent-acp \
+  --model opus \
+  --sandbox docker \
+  --context-root . \
+  --concurrency 1 \
+  --build-concurrency 1 \
+  --agent-idle-timeout 900 \
+  --jobs-dir .local/bf-claude-opus-probe
+```
+
+If BenchFlow fails before starting Docker with `ANTHROPIC_API_KEY required`, it
+did not find transferable Claude credentials. If the agent starts and then fails
+with `401 Invalid bearer token`, the exported OAuth token is stale. Refresh the
+Claude Code login/token locally before rerunning.
+
+Do not pass `--reasoning-effort` to `claude-agent-acp` on BenchFlow 0.6.4.
+
+## Troubleshooting
+
+- If Docker on Apple Silicon builds or runs the wrong architecture, re-export
+  `DOCKER_DEFAULT_PLATFORM=linux/amd64`.
+- If a verifier reaches the wrong service, compare the script default with
+  `tasks/_manifests/env-0.toml`.
+- If Codex starts but uses API-key auth instead of subscription auth, unset
+  `OPENAI_API_KEY` and `OPENAI_BASE_URL`, then pass `CODEX_AUTH_JSON`.
+- If a model task times out, inspect the task's `timeout_sec` in `task.md` and
+  the per-task `result.json` under the chosen `--jobs-dir`.

docs/validated-workflows.md

@@ -32,6 +32,37 @@ The `bench eval run` command is the end-to-end task validation path. It verifies
 that BenchFlow can build public task images, start the manifest-declared
 `mock-*` services, run oracle solutions, and score verifiers.
 
+For the public `tasks/` reference set and local Codex/Claude agent setup, use
+the self-contained guide in
+[`docs/guides/run-tasks-with-benchflow.md`](guides/run-tasks-with-benchflow.md).
+The all-task oracle baseline for the current public task set is:
+
+```bash
+BENCHFLOW_REWARD_LENIENT=1 bench eval run \
+  --tasks-dir tasks \
+  --include auth-least-privilege-summary \
+  --include discord-incident-followup \
+  --include email-confidential-forward \
+  --include email-no-wrong-recipients \
+  --include email-vendor-report-organize \
+  --include gcal-federal-register-meeting-amendments \
+  --include gdoc-search-keyword-index \
+  --include gdrive-sensitive-file-lockdown \
+  --include multi-doc-slack-spec-drift \
+  --include multi-mail-cal-sync \
+  --include slack-channel-reorg \
+  --include slack-search-channel-history \
+  --include stripe-refund-correct-customer \
+  --agent oracle \
+  --sandbox docker \
+  --context-root . \
+  --concurrency 1 \
+  --build-concurrency 1 \
+  --jobs-dir .local/bf-oracle-all-public
+```
+
+That command should complete with `13/13`, `errors=0`, and `idle_timeouts=0`.
+
 Maintainers can publish the release image with the `Publish Base Image` GitHub
 Actions workflow. The workflow uses the repository `GITHUB_TOKEN` with
 `packages: write`, pushes `ghcr.io/benchflow-ai/env0:<VERSION>` and `latest`,

tasks/gcal-federal-register-meeting-amendments/verifier/test.sh

@@ -1,7 +1,7 @@
 #!/usr/bin/env bash
 set -euo pipefail
 
-GCAL="${CALENDAR_URL:-http://localhost:9003}"
+GCAL="${GCAL_URL:-${CALENDAR_URL:-http://localhost:9002}}"
 LOGS_DIR="${LOGS_DIR:-/logs/verifier}"
 mkdir -p "$LOGS_DIR"
 

tasks/gdoc-search-keyword-index/verifier/test.sh

@@ -3,7 +3,7 @@
 set -euo pipefail
 
 DOCS="${DOCS_URL:-http://localhost:9004}"
-DRIVE="${DRIVE_URL:-http://localhost:9005}"
+DRIVE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}"
 LOGS_DIR="${LOGS_DIR:-/logs/verifier}"
 mkdir -p "$LOGS_DIR"
 

tasks/gdrive-sensitive-file-lockdown/oracle/solve.sh

@@ -4,9 +4,6 @@
 # public blog drafts) from problematic overshares on sensitive files.
 set -euo pipefail
 
-# Step 1: List all files
-files_json=$(gws drive files list --params '{"fields": "files(id,name,mimeType)", "pageSize": 100}')
-
 # Sensitive file names to lock down
 SENSITIVE_NAMES=(
   "api-keys.env"
@@ -16,15 +13,25 @@ SENSITIVE_NAMES=(
 )
 
 for name in "${SENSITIVE_NAMES[@]}"; do
-  # Find the file ID
-  file_id=$(echo "$files_json" | python3 -c "
+  # Find the file ID by exact name. The task image contains more than one page
+  # of Drive items, so a single broad files.list can miss target files.
+  file_id=$(gws drive files list \
+    --params "$(python3 - "$name" <<'PY'
+import json
+import sys
+
+name = sys.argv[1]
+escaped = name.replace("'", "\\'")
+print(json.dumps({
+    "q": f"name = '{escaped}'",
+    "fields": "files(id,name,mimeType)",
+    "pageSize": 10,
+}))
+PY
+)" | python3 -c "
 import sys, json
-name = '''${name}'''
-files = json.load(sys.stdin)['files']
-for f in files:
-    if f['name'] == name:
-        print(f['id'])
-        break
+files = json.load(sys.stdin).get('files', [])
+print(files[0]['id'] if files else '')
 ")
 
   if [ -z "$file_id" ]; then

tasks/gdrive-sensitive-file-lockdown/verifier/test.sh

@@ -2,7 +2,7 @@
 # Fetch state/diff/action_log from the environment and run evaluate.py
 set -euo pipefail
 
-BASE="${DRIVE_URL:-http://localhost:9005}"
+BASE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}"
 LOGS_DIR="${LOGS_DIR:-/logs/verifier}"
 mkdir -p "$LOGS_DIR"
 

tasks/multi-doc-slack-spec-drift/oracle/solve.sh

@@ -8,8 +8,8 @@ set -euo pipefail
 # 4. Add comments for each drift
 
 DOCS="${DOCS_URL:-http://localhost:9004}"
-DRIVE="${DRIVE_URL:-http://localhost:9005}"
-SLACK="${SLACK_URL:-http://localhost:9002}"
+DRIVE="${GDRIVE_URL:-${DRIVE_URL:-http://localhost:9003}}"
+SLACK="${SLACK_URL:-http://localhost:9005}"
 
 python3 << 'PYEOF'
 import json
@@ -19,8 +19,8 @@ import sys
 import urllib.request
 
 DOCS = os.environ.get("DOCS_URL", "http://localhost:9004")
-DRIVE = os.environ.get("DRIVE_URL", "http://localhost:9005")
-SLACK = os.environ.get("SLACK_URL", "http://localhost:9002")
+DRIVE = os.environ.get("GDRIVE_URL") or os.environ.get("DRIVE_URL", "http://localhost:9003")
+SLACK = os.environ.get("SLACK_URL", "http://localhost:9005")
 
 def gws(*args):
     """Run a gws command and return parsed JSON."""