Add the eval-ops toolkit used to drive the SWE-Marathon sweeps

eval-ops/PLAN.md

@@ -0,0 +1,125 @@
+# Standing orders — SWE-Marathon gpt-5.6-terra effort sweep
+
+**Read this first after any container restart.** This happened three times
+(2026-08-05 ~00:40Z, ~19:50Z, 2026-08-06 ~02:29Z), wiping `/home/user/terra-run`
+entirely — scripts, state, dataset copies — along with the babysit cron.
+
+**Root cause: the container is reclaimed on INACTIVITY.** Polling every 30
+minutes left gaps long enough to be reclaimed mid-run. The fix is a **20-minute
+heartbeat cron** that outputs a single letter and nothing else; with it, ordinary
+long-running loops survive and `run_forever.sh` can drive the whole fill. Keep
+that heartbeat alive — it is load-bearing, not cosmetic.
+
+The repos and `/home/user/oddish/.env` survive a wipe. Server-side trials keep
+running throughout; only the babysitting stops.
+
+**Recovery is a single command** — this whole directory is committed to the
+`oddish` repo, branch `claude/oddish-api-env-setup-f76jpp`, under `eval-ops/`:
+
+```bash
+cp -r /home/user/oddish/eval-ops/* /home/user/terra-run/     # or git checkout
+cd /home/user/terra-run
+for a in low medium high; do cp -r /home/user/swe-marathon/tasks ds-$a; done
+bash cycle.sh   # one idempotent pass: canary -> non-CUA dispatch -> throttled CUA fill
+
+```
+
+Then re-arm a 30-minute babysit cron (it is session-only and never survives).
+
+## The three experiments
+
+| Arm | Name | Experiment | `reasoning_effort` |
+|---|---|---|---|
+| LOW | `swem-terra-low` | `17b6f7d9` | `low` |
+| MEDIUM | `swem-terra-medium` | `c071229f` | `medium` |
+| HIGH | `swem-terra-high` | `a706e700` | `high` |
+
+Agent `codex`, model `openai/gpt-5.6-terra`, 20 SWE-Marathon v1.1 tasks each.
+Links: https://www.oddish.app/experiments/{17b6f7d9,c071229f,a706e700}
+
+## Charles's orders, in the order given
+
+1. **k=8 on the 16 non-CUA tasks, all three arms.** Done 2026-08-04T23:09Z —
+   48 cells verified at exactly 8. Completed clean: 384 trials, **0 infra**,
+   35 passes (HIGH 21 / MEDIUM 9 / LOW 5).
+2. **CUA tasks excluded** from that fill. They held baselines only: nop ×1 +
+   oracle ×1 per arm, 24 trials, all valid (nop 0.0 / oracle 1.0).
+3. **2026-08-05T18:12Z — "run 5 trials for all the cua trials rn"**, throttled
+   to the §3.6 cap of ≤10 concurrent. Reached ~23/60 before the second wipe.
+4. **2026-08-06T01:44Z — "finish the trials off. make everything 10/10".**
+   **Phase B, current:** every task, CUA and non-CUA, to **10 trials per cell**.
+   Non-CUA 8 → 10 (+96 trials, `dispatch.sh`); CUA → 10 (`cua_loop.sh`, still
+   throttled — the cap is a verifier-safety limit, not a target, and does
+   **not** relax in phase B).
+5. **Babysit every 30 minutes** throughout.
+
+`config.json` holds the live targets and phase; both scripts read it, so a
+phase change needs no relaunch.
+
+## Non-negotiables (runbook + hard-won)
+
+- `-e modal` always; `--override-memory-mb 65536` on every submission in every
+  arm (uniform, so memory never confounds the effort comparison). Verified
+  sufficient: **zero `exit 137` OOMs** across 384 trials at every effort level.
+- `--force` — preflight has a Rich-markup bug that flags every v1.1 task as
+  unjustified open internet. False positive.
+- `--no-baseline-gate` — the nop/oracle gate **is active** on this deployment
+  despite `gate_llm_on_baselines=False` in `config.py`. One flaky reward=0
+  oracle silently `skipped` agent trials in both MEDIUM and HIGH.
+- `--ae ODDISH_EVAL_NONCE=<unique>` — without it a resubmit after a delete hits
+  the 24 h sweep idempotency key, prints `Task submitted!`, and creates nothing.
+- **One task per command, serially, with backoff retries.** Multi-`-t` batch
+  submits return HTTP 500 and commit *zero* — verified repeatedly by re-query.
+  Singles 500 too, but clear on retry; this is why a fill pass takes ~30 min.
+- Sweeps are **target-based**: `--n-trials N` creates `N − existing`. Retries
+  are therefore safe and can never double-submit. This is what makes the whole
+  crash-recovery story work: just re-assert the target.
+- **Verify by data, never by exit code.** A submit that hits the tool timeout
+  has usually still committed — re-query rather than assuming.
+- Classify by `error_message`, never job status. Timeouts appear as prose
+  (`Agent execution timed out after N seconds`), not as `AgentTimeoutError` —
+  match both or a valid trial gets deleted as infra.
+- **Never** `--force-new-version`. If a submit reports a new task version, stop:
+  the dataset copy has drifted from the uploaded task. Rebuilt copies from a
+  fresh `swe-marathon` clone have been verified to reproduce
+  "unchanged, reusing version N".
+- Degenerate short trials are **reported, never auto-pruned** — in LOW, short
+  trajectories are the treatment effect, and pruning them biases pass@k.
+- Don't `pkill -f` on a pattern that also matches the calling shell's own
+  command line. It kills the caller. (Learned the hard way.)
+
+## CUA concurrency — SUPERSEDED, cap is now 25
+
+Charles, 2026-08-06: *"the concurrency can go up to 25. just queue up all of the
+trials for the cua. trust me."* The wave-by-wave filler is retired; `cua_dispatch.sh`
+queues every CUA cell straight to target and Oddish's own queue does the
+throttling. Do **not** re-impose the ≤10 waves. The original rationale is kept
+below for context, since it explains what the failure mode looks like if it ever
+does appear.
+
+## CUA throttle, original rationale (historical)
+
+The browser verifier runs on the shared platform `ANTHROPIC_API_KEY`, and each
+CUA `task.toml` hard-fails (no reward file → the trial *errors* rather than
+scores) if the grader dies for an infra reason. Over-parallelising destroys
+trials rather than merely queueing them. `nop` does not count — its verifier
+short-circuits in ~1.6 s. `cua_fill.py` refills only up to the headroom each
+pass, holds a single-writer lock so overlapping cycles can't both spend the
+same headroom, and aborts rather than submitting blind if any CUA read fails.
+
+Observed CUA behaviour: trials finish in ~15–30 min at $0.25–0.75 each, and
+have so far always produced a scored result — no `No reward file found` at a
+concurrency of 10, i.e. the cap is holding.
+
+## Files
+
+- `poll_all.py` — fetch + classify all 20 tasks → `state.json`
+- `babysit.py` — report → `babysit-latest.md` + `hb-fleet.txt`; `--delete`
+  removes infra, `--topup` re-asserts targets on short non-CUA cells only
+- `cua_fill.py` — one throttled CUA pass (config-driven, locked)
+- `cua_loop.sh` — repeats `cua_fill.py` until all 12 CUA cells hold target
+- `dispatch.sh` — 3 arms in parallel, 16 non-CUA tasks serially per arm
+- `config.json` — phase + targets + cap
+- `ds-low/`, `ds-medium/`, `ds-high/` — per-arm dataset copies so concurrent
+  submissions can't race on task files (not committed; rebuild from
+  `/home/user/swe-marathon/tasks`)

eval-ops/audit_exceptions.py

@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+"""Tally why trials actually failed, using Harbor's own classification.
+
+The authoritative answer is result.harbor_exception.exception_type -- Harbor
+names the failure itself. The error_message string is a red herring: it always
+opens with "Command failed (exit 1): opencode ..." regardless of cause, and its
+middle is elided server-side, so grepping it tells you very little.
+
+Also counts trials that carry a real verifier result despite the exception,
+since those solved the task and are being discarded by the validity rule.
+"""
+import json, subprocess, sys, collections
+
+ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish"
+EXP = "826d7d88"
+PENDING = ("pending", "queued", "running", "blocked", "preparing", "submitted",
+           "claimed", "in_progress", "initializing", "retrying")
+
+
+def api(args, tries=5):
+    for _ in range(tries):
+        try:
+            r = subprocess.run([ODDISH] + args, capture_output=True, text=True,
+                               timeout=600)
+            if r.returncode == 0 and r.stdout.strip():
+                return json.loads(r.stdout)
+        except Exception:
+            pass
+    return None
+
+
+d = api(["status", EXP, "--json"])
+if d is None:
+    sys.exit("experiment fetch FAILED")
+tasks = {t["id"]: t["name"] for t in d.get("tasks") or []}
+
+kinds = collections.Counter()
+by_task = collections.defaultdict(collections.Counter)
+scored_anyway = []      # exception raised, but the verifier still produced a reward
+missing = []
+
+for tid, tname in tasks.items():
+    td = api(["status", tid, "--json"])
+    if td is None:
+        missing.append(tname)
+        continue
+    for tr in td.get("trials") or []:
+        if tr.get("experiment_id") != EXP or tr.get("superseded_by_trial_id"):
+            continue
+        if (tr.get("status") or "").lower() in PENDING:
+            continue
+        res = tr.get("result") or {}
+        exc = (res.get("harbor_exception") or {}).get("exception_type")
+        if not exc:
+            kinds["(no exception -- clean run)"] += 1
+            continue
+        kinds[exc] += 1
+        by_task[tname][exc] += 1
+        rew = tr.get("reward")
+        main = (res.get("main") or {})
+        if rew is not None or main:
+            scored_anyway.append((tname, tr.get("id"), exc, rew,
+                                  main.get("passed"), main.get("total"),
+                                  (res.get("holdout") or {}).get("pass_rate")))
+
+if missing:
+    print(f"WARNING: {len(missing)} task(s) unreadable: {missing}")
+
+print("=== why trials ended, by Harbor's own exception_type ===")
+for k, v in kinds.most_common():
+    print(f"  {v:>4}  {k}")
+
+print("\n=== exceptions by task ===")
+for tname in sorted(by_task):
+    row = ", ".join(f"{k}={v}" for k, v in by_task[tname].most_common())
+    print(f"  {tname:<32} {row}")
+
+print(f"\n=== trials that raised an exception but STILL have a verifier result "
+      f"({len(scored_anyway)}) ===")
+for tname, tid, exc, rew, passed, total, hold in sorted(scored_anyway,
+                                                        key=lambda r: -(r[3] or 0)):
+    print(f"  {tid:<40} rew={rew} main={passed}/{total} holdout_rate={hold} {exc}")

eval-ops/babysit.py

@@ -0,0 +1,171 @@
+#!/usr/bin/env python3
+"""Babysit cycle for the three gpt-5.6-terra reasoning-effort experiments.
+
+One cycle = poll -> classify -> report -> (delete infra) -> (re-top-up).
+
+Targets live in config.json so they survive a container restart (see PLAN.md).
+Non-CUA cells are topped up directly; CUA cells are left to cua_fill.py, which
+respects the runbook 3.6 concurrency cap.
+
+Degenerate trials are REPORTED but never auto-deleted: in the LOW arm short
+trajectories are the treatment effect being measured, and systematically
+pruning them biases pass@k (runbook 4b).
+
+Usage:  babysit.py [--delete] [--topup]
+"""
+import json, subprocess, os, sys, statistics, collections
+
+HERE = "/home/user/terra-run"
+ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish"
+ARMS = ["LOW", "MEDIUM", "HIGH"]
+EXP_OF = {"LOW": "17b6f7d9", "MEDIUM": "c071229f", "HIGH": "a706e700"}
+CUA = {"excel-clone", "mastodon-clone", "s3-clone", "slack-clone"}
+
+CFG = json.load(open(f"{HERE}/config.json"))
+N_NONCUA, N_CUA, CUA_CAP = CFG["noncua_target"], CFG["cua_target"], CFG["cua_cap"]
+
+
+def load_env():
+    env = dict(os.environ)
+    for line in open("/home/user/oddish/.env"):
+        line = line.strip()
+        if line and not line.startswith("#") and "=" in line:
+            k, v = line.split("=", 1)
+            env[k.strip()] = v.strip().strip('"').strip("'")
+    return env
+
+
+def main():
+    do_delete, do_topup = "--delete" in sys.argv, "--topup" in sys.argv
+    env = load_env()
+    r = subprocess.run(["python3", f"{HERE}/poll_all.py"], capture_output=True,
+                       text=True, env=env, cwd=HERE, timeout=2400)
+    print(r.stdout.strip() or r.stderr.strip()[:500])
+    st = json.load(open(f"{HERE}/state.json"))
+
+    cells, now = st["cells"], st["polled_at"]
+    tasks = sorted({k.split("|")[1] for k in cells})
+    noncua = [t for t in tasks if t not in CUA]
+    cua = [t for t in tasks if t in CUA]
+    cua_live = sum(c["PENDING"] for k, c in cells.items()
+                   if k.split("|")[1] in CUA and k.split("|")[2] != "nop")
+
+    out = [f"# Babysit report — {now}", "",
+           f"Phase **{CFG['phase']}** — targets: non-CUA **{N_NONCUA}**/cell, "
+           f"CUA **{N_CUA}**/cell.", "",
+           f"CUA in-flight (judge-consuming): **{cua_live}** / cap {CUA_CAP} "
+           f"— {'OK' if cua_live <= CUA_CAP else 'OVER CAP'}", ""]
+
+    grand = collections.Counter()
+    for label, tlist, target in (("non-CUA", noncua, N_NONCUA), ("CUA", cua, N_CUA)):
+        for arm in ARMS:
+            rows = [(t, cells.get(f"{arm}|{t}|terra",
+                                  {"VALID": 0, "PENDING": 0, "INFRA": 0,
+                                   "OTHER": 0, "SKIPPED": 0, "pass": 0}))
+                    for t in tlist]
+            v = sum(c["VALID"] for _, c in rows)
+            p = sum(c["PENDING"] for _, c in rows)
+            i = sum(c["INFRA"] + c["OTHER"] + c["SKIPPED"] for _, c in rows)
+            ps = sum(c["pass"] for _, c in rows)
+            grand.update({"valid": v, "pending": p, "infra": i, "pass": ps,
+                          f"{label}_valid": v, f"{label}_pass": ps})
+            done = sum(1 for _, c in rows if c["VALID"] >= target)
+            out += [f"## {label} — {arm} ({EXP_OF[arm]})", "",
+                    f"valid={v}/{len(tlist)*target} pending={p} infra={i} "
+                    f"pass={ps} — {done}/{len(tlist)} tasks at target", "",
+                    "| task | valid | pending | infra | pass |", "|---|---|---|---|---|"]
+            for t, c in rows:
+                flag = "" if c["VALID"] >= target else " ⏳"
+                out.append(f"| `{t}`{flag} | {c['VALID']} | {c['PENDING']} | "
+                           f"{c['INFRA']+c['OTHER']+c['SKIPPED']} | {c['pass']} |")
+            out.append("")
+
+    out += [f"**Fleet:** valid={grand['valid']} pending={grand['pending']} "
+            f"infra={grand['infra']} pass={grand['pass']}", ""]
+
+    bad = (st["trials"].get("INFRA", []) + st["trials"].get("OTHER", [])
+           + st["trials"].get("SKIPPED", []))
+    if bad:
+        out += ["## Infra / skipped — delete + rerun", ""]
+        for tr in bad:
+            out.append(f"- `{tr['id']}` {tr['arm']}/{tr['task']}/{tr['agent']} "
+                       f"[{tr['status']}] {tr['err']}")
+        out.append("")
+
+    short = [tr for tr in st["trials"].get("VALID", [])
+             if tr["kind"] == "terra" and (tr["steps"] or 0) < 15]
+    if short:
+        out += ["## Degenerate candidates (<15 steps) — reported, not deleted", ""]
+        for tr in sorted(short, key=lambda x: x["steps"] or 0):
+            out.append(f"- `{tr['id']}` {tr['arm']}/{tr['task']} "
+                       f"steps={tr['steps']} reward={tr['reward']}")
+        out.append("")
+
+    allsteps = [tr["steps"] for tr in st["trials"].get("VALID", [])
+                if tr["kind"] == "terra" and tr["steps"]]
+    if allsteps:
+        out.append(f"Step distribution (valid terra, n={len(allsteps)}): "
+                   f"min={min(allsteps)} p50={int(statistics.median(allsteps))} "
+                   f"max={max(allsteps)}")
+    cost = sum(tr["cost"] or 0 for recs in st["trials"].values() for tr in recs
+               if tr["kind"] == "terra")
+    out.append(f"Terra spend so far: ${cost:,.2f}")
+
+    report = "\n".join(out) + "\n"
+    open(f"{HERE}/babysit-latest.md", "w").write(report)
+    print(report)
+    with open(f"{HERE}/hb-fleet.txt", "w") as f:
+        f.write(f"{now} | FLEET phase={CFG['phase']} | valid={grand['valid']} "
+                f"pending={grand['pending']} infra={grand['infra']} | "
+                f"cua_inflight={cua_live}/{CUA_CAP} | ${cost:,.0f}\n")
+
+    if do_delete and bad:
+        denv = dict(env)
+        denv["ODDISH_API_KEY"] = denv["ODDISH_ADMIN_API_KEY"]
+        ids = [tr["id"] for tr in bad]
+        for i in range(0, len(ids), 20):
+            chunk = ids[i:i + 20]
+            args = [ODDISH, "delete"] + sum([["-t", x] for x in chunk], []) + ["--json"]
+            rr = subprocess.run(args, capture_output=True, text=True, env=denv,
+                                timeout=600)
+            print(f"deleted {len(chunk)}: rc={rr.returncode} {rr.stdout[:200]}")
+
+    if do_topup:
+        # Only non-CUA here; CUA refills go through cua_fill.py's throttle.
+        shortc = []
+        for arm in ARMS:
+            for t in noncua:
+                c = cells.get(f"{arm}|{t}|terra")
+                held = sum(c[k] for k in ("VALID", "PENDING", "INFRA", "OTHER",
+                                          "SKIPPED")) if c else 0
+                if held < N_NONCUA:
+                    shortc.append((arm, t, N_NONCUA - held))
+        if not shortc:
+            print(f"topup: nothing short — all non-CUA cells hold {N_NONCUA}")
+        for arm, t, gap in shortc:
+            print(f"topup: {arm}/{t} short by {gap}")
+            for attempt in range(1, 6):
+                rr = subprocess.run(
+                    [ODDISH, "run", "-p", f"{HERE}/ds-{arm.lower()}", "-t", t,
+                     "-a", "codex", "-m", "openai/gpt-5.6-terra",
+                     "--n-trials", str(N_NONCUA), "-e", "modal",
+                     "-E", EXP_OF[arm], "--override-memory-mb", "65536",
+                     "--no-baseline-gate",
+                     "--ak", f"reasoning_effort={arm.lower()}",
+                     "--ae", f"ODDISH_EVAL_NONCE={arm}-{t}-{now}-{attempt}",
+                     "--background", "--force"],
+                    capture_output=True, text=True, env=env, timeout=400)
+                if "new version" in rr.stdout.lower():
+                    print(f"  ABORT {arm}/{t}: submit wants a NEW TASK VERSION")
+                    break
+                if "Task submitted!" in rr.stdout:
+                    line = [l.strip() for l in rr.stdout.splitlines() if "Trials:" in l]
+                    print(f"  ok {line[0] if line else ''}")
+                    break
+                print(f"  attempt {attempt} failed")
+                if attempt < 5:
+                    subprocess.run(["sleep", str(5 * attempt * attempt)])
+
+
+if __name__ == "__main__":
+    main()