From 6203df87a76201f60ee3760300bf5cdb0630e979 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 00:35:02 +0000 Subject: [PATCH 1/2] Add cloud session bootstrap for eval runbook inputs The eval runbooks are written against a laptop: a venv at a ~ path, agent CLIs on PATH, secrets in ~/.env. A Claude Code cloud session has none of those, so every one of those steps fails verbatim and each session re-derives the same setup by hand. Add a remote-only SessionStart hook that rebuilds what can be rebuilt -- syncs oddish/ with the documented --extra server, puts the venv on PATH, and bridges configured credentials into ~/.env in a marked block -- then reports one status line per runbook input so a session starts knowing what it has and which secrets are absent. It never invents a credential and always exits 0, so a partial bootstrap degrades a session instead of blocking it. Document the rest in docs/cloud-session-setup.md, since some inputs cannot be rebuilt from inside the container: secrets belong to the environment's variable config, and laptop-only scripts have to reach a repo before a session can see them. Record that oddish has no ~/.oddish credentials file -- config.py reads ODDISH_API_KEY/ODDISH_API_URL from the environment and nothing else -- so that hunt ends at the docs. The hook is not registered in .claude/settings.json here; enabling a hook that auto-runs for everyone who clones the repo is a call for a human to make, and the snippet to do it is in the doc. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NSorCmyqJqywkZF6cKku6n --- .claude/hooks/session-start.sh | 126 +++++++++++++++++++++++++++ docs/cloud-session-setup.md | 136 ++++++++++++++++++++++++++++++ docs/swe-marathon-eval-runbook.md | 5 ++ 3 files changed, 267 insertions(+) create mode 100755 .claude/hooks/session-start.sh create mode 100644 docs/cloud-session-setup.md diff --git a/.claude/hooks/session-start.sh b/.claude/hooks/session-start.sh new file mode 100755 index 000000000..48838eab3 --- /dev/null +++ b/.claude/hooks/session-start.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# SessionStart bootstrap for Claude Code on the web. +# +# Cloud sessions get a fresh container with the repo cloned and nothing else: +# no virtualenv, no agent CLIs, no ~/.env, no shell profile. This script +# reconstructs the pieces the eval runbooks assume, so a session can run +# `oddish` without a manual setup round-trip first. +# +# It never invents credentials. Secrets come from the environment's variable +# config (claude.ai/code -> Environments); this script only bridges them to the +# places local tooling looks for them, and reports what is missing. +# +# See docs/cloud-session-setup.md. +set -uo pipefail + +# Local machines already have a venv, a shell profile, and the agent CLIs on +# PATH. Only the disposable remote container needs rebuilding. +if [ "${CLAUDE_CODE_REMOTE:-}" != "true" ]; then + exit 0 +fi + +REPO_ROOT="${CLAUDE_PROJECT_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}" +ENV_HOME="${HOME:-/root}" + +# Credentials bridged into ~/.env for scripts that `source` it. AWS is +# deliberately absent: the log-bucket creds are short-lived STS tokens with +# their own refresh cycle (runbook section 6), not session-long config. +BRIDGED_VARS=( + ODDISH_API_KEY + ODDISH_API_URL + XAI_API_KEY + XAI_API_KEYS + ANTHROPIC_API_KEY + OPENAI_API_KEY + META_API_KEY +) + +status_lines=() +note() { status_lines+=("$1"); } + +# --- oddish CLI ------------------------------------------------------------ +# Replaces the local ~/oddish/oddish/.venv that runbooks reference by path. +# --extra server matches the documented setup (AGENTS.md, "Local Development"); +# without it the test suite cannot import sqlalchemy. +venv_bin="$REPO_ROOT/oddish/.venv/bin" +if command -v uv >/dev/null 2>&1; then + sync_log="$(cd "$REPO_ROOT/oddish" && uv sync --frozen --extra server 2>&1)" || \ + sync_log="$(cd "$REPO_ROOT/oddish" && uv sync --extra server 2>&1)" + if [ -x "$venv_bin/oddish" ]; then + note "oddish CLI ok $venv_bin/oddish" + else + note "oddish CLI FAILED uv sync did not produce the venv" + printf '%s\n' "$sync_log" | tail -20 >&2 + fi +else + note "oddish CLI FAILED uv not on PATH" +fi + +# --- grok CLI -------------------------------------------------------------- +# Reported, not installed: pulling and running a remote installer at session +# start is a network dependency the session cannot audit. Install it in-session +# when a task needs it -- docs/cloud-session-setup.md has the command. +if [ -x "$ENV_HOME/.grok/bin/grok" ]; then + note "grok CLI ok $ENV_HOME/.grok/bin/grok" +else + note "grok CLI absent see docs/cloud-session-setup.md to install" +fi + +# --- ~/.env ---------------------------------------------------------------- +# Bridges configured secrets to the file local scripts source. Rewrites only +# its own marker block so anything hand-added during the session survives. +env_file="$ENV_HOME/.env" +begin="# >>> oddish cloud bootstrap >>>" +end="# <<< oddish cloud bootstrap <<<" +bridged=() +block="$begin"$'\n'"# Generated at session start from the environment's variable config." +for var in "${BRIDGED_VARS[@]}"; do + if [ -n "${!var:-}" ]; then + block+=$'\n'"export $var=${!var@Q}" + bridged+=("$var") + fi +done +block+=$'\n'"$end" + +if [ -f "$env_file" ] && grep -qF "$begin" "$env_file" 2>/dev/null; then + kept="$(awk -v b="$begin" -v e="$end" \ + 'index($0,b){s=1} !s{print} index($0,e){s=0}' "$env_file")" +else + kept="$(cat "$env_file" 2>/dev/null)" +fi +umask 077 +{ [ -n "$kept" ] && printf '%s\n' "$kept"; printf '%s\n' "$block"; } > "$env_file" +chmod 600 "$env_file" + +if [ ${#bridged[@]} -gt 0 ]; then + note "~/.env ok ${#bridged[@]} var(s): ${bridged[*]}" +else + note "~/.env empty no known credentials set on this environment" +fi + +# --- session env ----------------------------------------------------------- +if [ -n "${CLAUDE_ENV_FILE:-}" ]; then + { + [ -d "$venv_bin" ] && echo "export PATH=\"$venv_bin:\$PATH\"" + echo "export PATH=\"$ENV_HOME/.grok/bin:$ENV_HOME/.local/bin:\$PATH\"" + } >> "$CLAUDE_ENV_FILE" +fi + +# --- report ---------------------------------------------------------------- +# SessionStart stdout joins the session context, so the agent starts knowing +# what it has. Names and states only -- never values. +echo "Cloud bootstrap (docs/cloud-session-setup.md):" +printf ' %s\n' "${status_lines[@]}" + +missing=() +for var in ODDISH_API_KEY XAI_API_KEY; do + [ -z "${!var:-}" ] && missing+=("$var") +done +if [ ${#missing[@]} -gt 0 ]; then + echo " missing secrets: ${missing[*]}" + echo " -> set them on this environment at claude.ai/code (Environments ->" + echo " this environment -> environment variables), then start a new session." +fi + +# Always succeed: a partial bootstrap should degrade the session, not block it. +exit 0 diff --git a/docs/cloud-session-setup.md b/docs/cloud-session-setup.md new file mode 100644 index 000000000..936f948ec --- /dev/null +++ b/docs/cloud-session-setup.md @@ -0,0 +1,136 @@ +# Cloud session setup (Claude Code on the web) + +How to give a cloud Claude Code session the things the eval runbooks assume. +The short version: **secrets come from the environment's variable config, code +comes from attached repos, and everything else is rebuilt by the SessionStart +hook.** Nothing is copied from a laptop. + +## What a fresh cloud container actually has + +A cloud session runs in an ephemeral container: this repo is cloned fresh, the +container is reclaimed when the session ends, and anything not committed is +lost. It starts with no virtualenv, no `~/.env`, no shell profile, no agent +CLIs, and no credentials beyond what the environment injects. + +That is why runbook steps written against a laptop fail verbatim. Most of them +do not need porting — they need translating: + +| Runbook input (laptop) | Cloud equivalent | +| ----------------------------------------- | ----------------------------------------------------------------------- | +| `~/oddish/oddish/.venv/bin/oddish` | `/oddish/.venv/bin/oddish` — built by the hook, and on `PATH` | +| `~/.grok/bin/grok` | installed in-session on demand (see below) | +| `XAI_API_KEY` via `~/.env` | set on the environment; the hook writes it back into `~/.env` | +| `~/.oddish` credentials | **does not exist** — see below | +| `~/cyberpipeline/*.sh` and similar | must live in a git repo and be attached to the session | + +### There is no `~/.oddish` credentials file + +The CLI reads credentials from the environment only — `ODDISH_API_KEY` and +`ODDISH_API_URL`, resolved in `oddish/src/oddish/cli/config.py` +(`get_api_key`, `get_api_url`). There is no credentials file, no `oddish login`, +and no dotfile to copy. Setting the two environment variables is the whole of +CLI auth, locally and in the cloud alike. + +## 1. Secrets: set them on the environment + +Environment variables are configured per environment at +[claude.ai/code](https://claude.ai/code) → Environments → *(your environment)* → +environment variables. They are injected into every session that environment +starts, so this is a one-time setup rather than a per-session step. + +Set what the work needs: + +| Variable | Needed for | +| ------------------- | ------------------------------------------------------------ | +| `ODDISH_API_KEY` | every `oddish` command that talks to the hosted API | +| `ODDISH_API_URL` | only to target a non-default API (preview, self-hosted) | +| `XAI_API_KEY` | the grok CLI, and any script that authenticates to xAI | +| `ANTHROPIC_API_KEY` | CUA verifiers on the open-internet tasks | +| `OPENAI_API_KEY` / `META_API_KEY` | vendor routes that read them | + +Two things these variables are *not* for: + +- **Trial credentials.** A vendor key used by an agent inside a trial must be a + **Modal secret** on the worker function, not a session variable — see the + runbook's prereqs. A key set here is available to the session driving the + eval, not to the sandboxes running it. +- **The log-bucket export.** Those AWS credentials are short-lived STS tokens + (~1h) with their own refresh cycle, so the hook deliberately leaves `AWS_*` + alone. Export them per-session as the runbook describes. + +## 2. Code: attach the repo it lives in + +A cloud session can only see repositories attached to it. Scripts that live +only on a laptop — a `~/cyberpipeline` working directory, `author_prompt.sh`, +`ship.sh` — are unreachable no matter how the environment is configured. Push +them to a repo first; then a session can be given access to it, and can clone it +alongside this one. + +The eval runbook's background-agent prompt already assumes this: it asks for +access to `abundant-ai/oddish`, `abundant-ai/harbor`, and +`abundant-ai/swe-marathon`. Add whichever repo holds the pipeline scripts to +that list. + +## 3. Everything else: the SessionStart hook + +`.claude/hooks/session-start.sh` runs when a cloud session starts and: + +- runs `uv sync --frozen --extra server` in `oddish/`, producing + `oddish/.venv/bin/oddish` (the `--extra server` matches AGENTS.md; without it + the test suite cannot import `sqlalchemy`) +- puts that venv, `~/.grok/bin`, and `~/.local/bin` on `PATH` +- writes the configured credentials into `~/.env` for scripts that `source` it, + in a marked block so anything else in the file survives +- prints a status line per input, and names any missing secret + +It is remote-only (`CLAUDE_CODE_REMOTE`), idempotent, and always exits 0 — a +partial bootstrap degrades a session rather than blocking it. It never invents +a credential: if a variable is not set on the environment, the hook says so. + +To enable it, register it in `.claude/settings.json`: + +```json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "type": "command", + "command": "$CLAUDE_PROJECT_DIR/.claude/hooks/session-start.sh" + } + ] + } + ] + } +} +``` + +The hook takes effect for sessions started after that lands on the default +branch. + +### Installing the grok CLI + +The hook reports the grok CLI but does not install it: it is a network install +most sessions never need. When a task does need it: + +```bash +curl -fsSL https://x.ai/cli/install.sh -o /tmp/grok-install.sh +bash /tmp/grok-install.sh +export PATH="$HOME/.grok/bin:$PATH" +``` + +This is the same installer `OddishGrokBuildAgent.install()` runs inside Harbor +sandboxes (`oddish/src/oddish/workers/agents/grok_build.py`). Note the +distinction: that in-sandbox install is what trials use, and it happens whether +or not the CLI is present in the session. You only need it in the session +itself if *you* are driving grok directly. + +## Verifying + +The hook's own output is the check — it prints one line per input at session +start. To re-run it by hand: + +```bash +CLAUDE_CODE_REMOTE=true .claude/hooks/session-start.sh +``` diff --git a/docs/swe-marathon-eval-runbook.md b/docs/swe-marathon-eval-runbook.md index 02874b9e1..87096c2ae 100644 --- a/docs/swe-marathon-eval-runbook.md +++ b/docs/swe-marathon-eval-runbook.md @@ -32,6 +32,11 @@ Harbor `ExceptionGroup`) is **not** valid; delete and rerun those. isn't wired where the client reads it (e.g. LiteLLM's `openai/` provider reads `OPENAI_API_KEY`, not a vendor-specific name). 4. CLI points at hosted Oddish: `ODDISH_API_URL` + `ODDISH_API_KEY`. +5. Running this from a Claude Code cloud session? The steps above assume a + laptop: a venv at a `~` path, agent CLIs on `PATH`, secrets in `~/.env`. A + fresh cloud container has none of those. See + `docs/cloud-session-setup.md` for how each one is provisioned there — and + note that step 4 is the *whole* of CLI auth: there is no credentials file. ## 1. Confirm the current 20 tasks and execution classes From 09ee550985f4459552db7330b16471cab8a29232 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 12 Aug 2026 00:45:41 +0000 Subject: [PATCH 2/2] Fix PATH precedence and duplication in cloud bootstrap hook Two defects found in review, both confirmed in a container. The two PATH exports were appended in sequence, so the second prepended onto the result of the first and ~/.local/bin landed ahead of the venv. That directory ships its own pytest, ruff, black, and mypy -- exactly the tools this repo tests and lints with -- so `pytest` resolved to the user-local copy instead of the project's. Emit a single export with the venv first. SessionStart also fires on resume, clear, and compact, and CLAUDE_ENV_FILE persists across those firings, so the unguarded append stacked another PATH prefix every time a session compacted. Write the line at most once; three consecutive firings now leave one. The uv sync still runs on every firing. That is deliberate: it costs about a second once the cache is warm, and it repairs a venv that is broken or partially installed -- the state that produced the ModuleNotFoundError which pinned --extra server in the first place. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NSorCmyqJqywkZF6cKku6n --- .claude/hooks/session-start.sh | 16 ++++++++++++---- docs/cloud-session-setup.md | 11 ++++++++++- 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/.claude/hooks/session-start.sh b/.claude/hooks/session-start.sh index 48838eab3..266462554 100755 --- a/.claude/hooks/session-start.sh +++ b/.claude/hooks/session-start.sh @@ -99,11 +99,19 @@ else fi # --- session env ----------------------------------------------------------- +# One export, venv first: ~/.local/bin carries its own pytest/ruff/black/mypy, +# which would shadow the venv's if it were prepended afterwards. +# +# Written at most once. SessionStart also fires on resume, clear, and compact, +# and CLAUDE_ENV_FILE persists across those firings, so an unguarded append +# would stack a duplicate prefix onto PATH every time the session compacted. if [ -n "${CLAUDE_ENV_FILE:-}" ]; then - { - [ -d "$venv_bin" ] && echo "export PATH=\"$venv_bin:\$PATH\"" - echo "export PATH=\"$ENV_HOME/.grok/bin:$ENV_HOME/.local/bin:\$PATH\"" - } >> "$CLAUDE_ENV_FILE" + path_prefix="" + [ -d "$venv_bin" ] && path_prefix="$venv_bin:" + path_line="export PATH=\"$path_prefix$ENV_HOME/.grok/bin:$ENV_HOME/.local/bin:\$PATH\"" + if ! grep -qxF "$path_line" "$CLAUDE_ENV_FILE" 2>/dev/null; then + printf '%s\n' "$path_line" >> "$CLAUDE_ENV_FILE" + fi fi # --- report ---------------------------------------------------------------- diff --git a/docs/cloud-session-setup.md b/docs/cloud-session-setup.md index 936f948ec..beba586f2 100644 --- a/docs/cloud-session-setup.md +++ b/docs/cloud-session-setup.md @@ -78,7 +78,9 @@ that list. - runs `uv sync --frozen --extra server` in `oddish/`, producing `oddish/.venv/bin/oddish` (the `--extra server` matches AGENTS.md; without it the test suite cannot import `sqlalchemy`) -- puts that venv, `~/.grok/bin`, and `~/.local/bin` on `PATH` +- puts that venv, `~/.grok/bin`, and `~/.local/bin` on `PATH`, venv first — + `~/.local/bin` ships its own `pytest`/`ruff`/`black`/`mypy`, which would + otherwise shadow the project's - writes the configured credentials into `~/.env` for scripts that `source` it, in a marked block so anything else in the file survives - prints a status line per input, and names any missing secret @@ -87,6 +89,13 @@ It is remote-only (`CLAUDE_CODE_REMOTE`), idempotent, and always exits 0 — a partial bootstrap degrades a session rather than blocking it. It never invents a credential: if a variable is not set on the environment, the hook says so. +Idempotency matters here because `SessionStart` fires on `resume`, `clear`, and +`compact` as well as on startup, and a long eval session compacts repeatedly. +Each firing rewrites only its own `~/.env` block and writes the `PATH` line at +most once, so nothing accumulates. The `uv sync` does re-run every time; that is +deliberate and costs about a second once the cache is warm, and it repairs a +venv that has been broken or partially installed mid-session. + To enable it, register it in `.claude/settings.json`: ```json