From 4e06335a6a0a9e570704aca6272e917e512a594f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 01:32:20 +0000 Subject: [PATCH 1/5] Add the eval-ops toolkit used to drive the SWE-Marathon sweeps Operational scripts for running the SWE-Marathon evals end to end, collected here so they survive a container recycle and so the next run does not have to rediscover the same gotchas. Nothing here is imported by the CLI, the backend, or the frontend; it is a standalone toolkit under eval-ops/. What it covers: - Filling an experiment to N trials per task. Sweeps are target-based, so --n-trials N creates N - existing and re-running a pass is never a double submit. even_fill.py tops up fewest-held-first under a global in-flight cap and a per-pass batch size, so all 20 tasks advance together rather than the scheduler draining one; even_loop.sh repeats it until every cell is at target. - Throttling to what the providers actually allow. Gemini's input-token quota is shared fleet-wide: at 34-35 concurrent trials every completion came back 429 RESOURCE_EXHAUSTED, and each 429 makes opencode retry with the full context re-sent, inflating usage further. At ~15 concurrent, completions come back clean. The CUA-verifier tasks have their own cap, handled by cua_fill.py. - Classifying outcomes by error_message rather than job status, because infra failures routinely report status=success with reward=0. poll_all.py and oc_poll.py both use that rule; "retrying" is a pending state, not a failure, so a cleanup pass does not destroy trials that were about to succeed. - passk.py computes pass@k with the same estimator as the frontend's pass-at-k.ts, scoring only trials that reached a real terminal state, and refuses to score an experiment whose tasks it could not all read. - clear_broken.py deletes only zero-token infra failures, keeps trials that consumed tokens, and skips anything still in flight. Agent flags are passed explicitly everywhere: opencode's `variant` is a CliFlag with no default, so omitting it silently runs a different configuration than the arm being reproduced. --- eval-ops/PLAN.md | 125 ++++++++++++++++++++++ eval-ops/babysit.py | 171 +++++++++++++++++++++++++++++ eval-ops/clear_broken.py | 123 +++++++++++++++++++++ eval-ops/config.json | 7 ++ eval-ops/cua_dispatch.sh | 50 +++++++++ eval-ops/cua_fill.py | 196 ++++++++++++++++++++++++++++++++++ eval-ops/cua_loop.sh | 23 ++++ eval-ops/cycle.sh | 57 ++++++++++ eval-ops/dispatch.sh | 60 +++++++++++ eval-ops/even_fill.py | 147 +++++++++++++++++++++++++ eval-ops/even_loop.sh | 45 ++++++++ eval-ops/oc_poll.py | 67 ++++++++++++ eval-ops/opencode_dispatch.sh | 32 ++++++ eval-ops/passk.py | 171 +++++++++++++++++++++++++++++ eval-ops/poll_all.py | 144 +++++++++++++++++++++++++ eval-ops/run_forever.sh | 63 +++++++++++ 16 files changed, 1481 insertions(+) create mode 100644 eval-ops/PLAN.md create mode 100644 eval-ops/babysit.py create mode 100644 eval-ops/clear_broken.py create mode 100644 eval-ops/config.json create mode 100755 eval-ops/cua_dispatch.sh create mode 100644 eval-ops/cua_fill.py create mode 100755 eval-ops/cua_loop.sh create mode 100755 eval-ops/cycle.sh create mode 100755 eval-ops/dispatch.sh create mode 100644 eval-ops/even_fill.py create mode 100755 eval-ops/even_loop.sh create mode 100644 eval-ops/oc_poll.py create mode 100755 eval-ops/opencode_dispatch.sh create mode 100644 eval-ops/passk.py create mode 100644 eval-ops/poll_all.py create mode 100755 eval-ops/run_forever.sh diff --git a/eval-ops/PLAN.md b/eval-ops/PLAN.md new file mode 100644 index 000000000..dca728b91 --- /dev/null +++ b/eval-ops/PLAN.md @@ -0,0 +1,125 @@ +# Standing orders — SWE-Marathon gpt-5.6-terra effort sweep + +**Read this first after any container restart.** This happened three times +(2026-08-05 ~00:40Z, ~19:50Z, 2026-08-06 ~02:29Z), wiping `/home/user/terra-run` +entirely — scripts, state, dataset copies — along with the babysit cron. + +**Root cause: the container is reclaimed on INACTIVITY.** Polling every 30 +minutes left gaps long enough to be reclaimed mid-run. The fix is a **20-minute +heartbeat cron** that outputs a single letter and nothing else; with it, ordinary +long-running loops survive and `run_forever.sh` can drive the whole fill. Keep +that heartbeat alive — it is load-bearing, not cosmetic. + +The repos and `/home/user/oddish/.env` survive a wipe. Server-side trials keep +running throughout; only the babysitting stops. + +**Recovery is a single command** — this whole directory is committed to the +`oddish` repo, branch `claude/oddish-api-env-setup-f76jpp`, under `eval-ops/`: + +```bash +cp -r /home/user/oddish/eval-ops/* /home/user/terra-run/ # or git checkout +cd /home/user/terra-run +for a in low medium high; do cp -r /home/user/swe-marathon/tasks ds-$a; done +bash cycle.sh # one idempotent pass: canary -> non-CUA dispatch -> throttled CUA fill + +``` + +Then re-arm a 30-minute babysit cron (it is session-only and never survives). + +## The three experiments + +| Arm | Name | Experiment | `reasoning_effort` | +|---|---|---|---| +| LOW | `swem-terra-low` | `17b6f7d9` | `low` | +| MEDIUM | `swem-terra-medium` | `c071229f` | `medium` | +| HIGH | `swem-terra-high` | `a706e700` | `high` | + +Agent `codex`, model `openai/gpt-5.6-terra`, 20 SWE-Marathon v1.1 tasks each. +Links: https://www.oddish.app/experiments/{17b6f7d9,c071229f,a706e700} + +## Charles's orders, in the order given + +1. **k=8 on the 16 non-CUA tasks, all three arms.** Done 2026-08-04T23:09Z — + 48 cells verified at exactly 8. Completed clean: 384 trials, **0 infra**, + 35 passes (HIGH 21 / MEDIUM 9 / LOW 5). +2. **CUA tasks excluded** from that fill. They held baselines only: nop ×1 + + oracle ×1 per arm, 24 trials, all valid (nop 0.0 / oracle 1.0). +3. **2026-08-05T18:12Z — "run 5 trials for all the cua trials rn"**, throttled + to the §3.6 cap of ≤10 concurrent. Reached ~23/60 before the second wipe. +4. **2026-08-06T01:44Z — "finish the trials off. make everything 10/10".** + **Phase B, current:** every task, CUA and non-CUA, to **10 trials per cell**. + Non-CUA 8 → 10 (+96 trials, `dispatch.sh`); CUA → 10 (`cua_loop.sh`, still + throttled — the cap is a verifier-safety limit, not a target, and does + **not** relax in phase B). +5. **Babysit every 30 minutes** throughout. + +`config.json` holds the live targets and phase; both scripts read it, so a +phase change needs no relaunch. + +## Non-negotiables (runbook + hard-won) + +- `-e modal` always; `--override-memory-mb 65536` on every submission in every + arm (uniform, so memory never confounds the effort comparison). Verified + sufficient: **zero `exit 137` OOMs** across 384 trials at every effort level. +- `--force` — preflight has a Rich-markup bug that flags every v1.1 task as + unjustified open internet. False positive. +- `--no-baseline-gate` — the nop/oracle gate **is active** on this deployment + despite `gate_llm_on_baselines=False` in `config.py`. One flaky reward=0 + oracle silently `skipped` agent trials in both MEDIUM and HIGH. +- `--ae ODDISH_EVAL_NONCE=` — without it a resubmit after a delete hits + the 24 h sweep idempotency key, prints `Task submitted!`, and creates nothing. +- **One task per command, serially, with backoff retries.** Multi-`-t` batch + submits return HTTP 500 and commit *zero* — verified repeatedly by re-query. + Singles 500 too, but clear on retry; this is why a fill pass takes ~30 min. +- Sweeps are **target-based**: `--n-trials N` creates `N − existing`. Retries + are therefore safe and can never double-submit. This is what makes the whole + crash-recovery story work: just re-assert the target. +- **Verify by data, never by exit code.** A submit that hits the tool timeout + has usually still committed — re-query rather than assuming. +- Classify by `error_message`, never job status. Timeouts appear as prose + (`Agent execution timed out after N seconds`), not as `AgentTimeoutError` — + match both or a valid trial gets deleted as infra. +- **Never** `--force-new-version`. If a submit reports a new task version, stop: + the dataset copy has drifted from the uploaded task. Rebuilt copies from a + fresh `swe-marathon` clone have been verified to reproduce + "unchanged, reusing version N". +- Degenerate short trials are **reported, never auto-pruned** — in LOW, short + trajectories are the treatment effect, and pruning them biases pass@k. +- Don't `pkill -f` on a pattern that also matches the calling shell's own + command line. It kills the caller. (Learned the hard way.) + +## CUA concurrency — SUPERSEDED, cap is now 25 + +Charles, 2026-08-06: *"the concurrency can go up to 25. just queue up all of the +trials for the cua. trust me."* The wave-by-wave filler is retired; `cua_dispatch.sh` +queues every CUA cell straight to target and Oddish's own queue does the +throttling. Do **not** re-impose the ≤10 waves. The original rationale is kept +below for context, since it explains what the failure mode looks like if it ever +does appear. + +## CUA throttle, original rationale (historical) + +The browser verifier runs on the shared platform `ANTHROPIC_API_KEY`, and each +CUA `task.toml` hard-fails (no reward file → the trial *errors* rather than +scores) if the grader dies for an infra reason. Over-parallelising destroys +trials rather than merely queueing them. `nop` does not count — its verifier +short-circuits in ~1.6 s. `cua_fill.py` refills only up to the headroom each +pass, holds a single-writer lock so overlapping cycles can't both spend the +same headroom, and aborts rather than submitting blind if any CUA read fails. + +Observed CUA behaviour: trials finish in ~15–30 min at $0.25–0.75 each, and +have so far always produced a scored result — no `No reward file found` at a +concurrency of 10, i.e. the cap is holding. + +## Files + +- `poll_all.py` — fetch + classify all 20 tasks → `state.json` +- `babysit.py` — report → `babysit-latest.md` + `hb-fleet.txt`; `--delete` + removes infra, `--topup` re-asserts targets on short non-CUA cells only +- `cua_fill.py` — one throttled CUA pass (config-driven, locked) +- `cua_loop.sh` — repeats `cua_fill.py` until all 12 CUA cells hold target +- `dispatch.sh` — 3 arms in parallel, 16 non-CUA tasks serially per arm +- `config.json` — phase + targets + cap +- `ds-low/`, `ds-medium/`, `ds-high/` — per-arm dataset copies so concurrent + submissions can't race on task files (not committed; rebuild from + `/home/user/swe-marathon/tasks`) diff --git a/eval-ops/babysit.py b/eval-ops/babysit.py new file mode 100644 index 000000000..ca78a05cd --- /dev/null +++ b/eval-ops/babysit.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +"""Babysit cycle for the three gpt-5.6-terra reasoning-effort experiments. + +One cycle = poll -> classify -> report -> (delete infra) -> (re-top-up). + +Targets live in config.json so they survive a container restart (see PLAN.md). +Non-CUA cells are topped up directly; CUA cells are left to cua_fill.py, which +respects the runbook 3.6 concurrency cap. + +Degenerate trials are REPORTED but never auto-deleted: in the LOW arm short +trajectories are the treatment effect being measured, and systematically +pruning them biases pass@k (runbook 4b). + +Usage: babysit.py [--delete] [--topup] +""" +import json, subprocess, os, sys, statistics, collections + +HERE = "/home/user/terra-run" +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +ARMS = ["LOW", "MEDIUM", "HIGH"] +EXP_OF = {"LOW": "17b6f7d9", "MEDIUM": "c071229f", "HIGH": "a706e700"} +CUA = {"excel-clone", "mastodon-clone", "s3-clone", "slack-clone"} + +CFG = json.load(open(f"{HERE}/config.json")) +N_NONCUA, N_CUA, CUA_CAP = CFG["noncua_target"], CFG["cua_target"], CFG["cua_cap"] + + +def load_env(): + env = dict(os.environ) + for line in open("/home/user/oddish/.env"): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + env[k.strip()] = v.strip().strip('"').strip("'") + return env + + +def main(): + do_delete, do_topup = "--delete" in sys.argv, "--topup" in sys.argv + env = load_env() + r = subprocess.run(["python3", f"{HERE}/poll_all.py"], capture_output=True, + text=True, env=env, cwd=HERE, timeout=2400) + print(r.stdout.strip() or r.stderr.strip()[:500]) + st = json.load(open(f"{HERE}/state.json")) + + cells, now = st["cells"], st["polled_at"] + tasks = sorted({k.split("|")[1] for k in cells}) + noncua = [t for t in tasks if t not in CUA] + cua = [t for t in tasks if t in CUA] + cua_live = sum(c["PENDING"] for k, c in cells.items() + if k.split("|")[1] in CUA and k.split("|")[2] != "nop") + + out = [f"# Babysit report — {now}", "", + f"Phase **{CFG['phase']}** — targets: non-CUA **{N_NONCUA}**/cell, " + f"CUA **{N_CUA}**/cell.", "", + f"CUA in-flight (judge-consuming): **{cua_live}** / cap {CUA_CAP} " + f"— {'OK' if cua_live <= CUA_CAP else 'OVER CAP'}", ""] + + grand = collections.Counter() + for label, tlist, target in (("non-CUA", noncua, N_NONCUA), ("CUA", cua, N_CUA)): + for arm in ARMS: + rows = [(t, cells.get(f"{arm}|{t}|terra", + {"VALID": 0, "PENDING": 0, "INFRA": 0, + "OTHER": 0, "SKIPPED": 0, "pass": 0})) + for t in tlist] + v = sum(c["VALID"] for _, c in rows) + p = sum(c["PENDING"] for _, c in rows) + i = sum(c["INFRA"] + c["OTHER"] + c["SKIPPED"] for _, c in rows) + ps = sum(c["pass"] for _, c in rows) + grand.update({"valid": v, "pending": p, "infra": i, "pass": ps, + f"{label}_valid": v, f"{label}_pass": ps}) + done = sum(1 for _, c in rows if c["VALID"] >= target) + out += [f"## {label} — {arm} ({EXP_OF[arm]})", "", + f"valid={v}/{len(tlist)*target} pending={p} infra={i} " + f"pass={ps} — {done}/{len(tlist)} tasks at target", "", + "| task | valid | pending | infra | pass |", "|---|---|---|---|---|"] + for t, c in rows: + flag = "" if c["VALID"] >= target else " ⏳" + out.append(f"| `{t}`{flag} | {c['VALID']} | {c['PENDING']} | " + f"{c['INFRA']+c['OTHER']+c['SKIPPED']} | {c['pass']} |") + out.append("") + + out += [f"**Fleet:** valid={grand['valid']} pending={grand['pending']} " + f"infra={grand['infra']} pass={grand['pass']}", ""] + + bad = (st["trials"].get("INFRA", []) + st["trials"].get("OTHER", []) + + st["trials"].get("SKIPPED", [])) + if bad: + out += ["## Infra / skipped — delete + rerun", ""] + for tr in bad: + out.append(f"- `{tr['id']}` {tr['arm']}/{tr['task']}/{tr['agent']} " + f"[{tr['status']}] {tr['err']}") + out.append("") + + short = [tr for tr in st["trials"].get("VALID", []) + if tr["kind"] == "terra" and (tr["steps"] or 0) < 15] + if short: + out += ["## Degenerate candidates (<15 steps) — reported, not deleted", ""] + for tr in sorted(short, key=lambda x: x["steps"] or 0): + out.append(f"- `{tr['id']}` {tr['arm']}/{tr['task']} " + f"steps={tr['steps']} reward={tr['reward']}") + out.append("") + + allsteps = [tr["steps"] for tr in st["trials"].get("VALID", []) + if tr["kind"] == "terra" and tr["steps"]] + if allsteps: + out.append(f"Step distribution (valid terra, n={len(allsteps)}): " + f"min={min(allsteps)} p50={int(statistics.median(allsteps))} " + f"max={max(allsteps)}") + cost = sum(tr["cost"] or 0 for recs in st["trials"].values() for tr in recs + if tr["kind"] == "terra") + out.append(f"Terra spend so far: ${cost:,.2f}") + + report = "\n".join(out) + "\n" + open(f"{HERE}/babysit-latest.md", "w").write(report) + print(report) + with open(f"{HERE}/hb-fleet.txt", "w") as f: + f.write(f"{now} | FLEET phase={CFG['phase']} | valid={grand['valid']} " + f"pending={grand['pending']} infra={grand['infra']} | " + f"cua_inflight={cua_live}/{CUA_CAP} | ${cost:,.0f}\n") + + if do_delete and bad: + denv = dict(env) + denv["ODDISH_API_KEY"] = denv["ODDISH_ADMIN_API_KEY"] + ids = [tr["id"] for tr in bad] + for i in range(0, len(ids), 20): + chunk = ids[i:i + 20] + args = [ODDISH, "delete"] + sum([["-t", x] for x in chunk], []) + ["--json"] + rr = subprocess.run(args, capture_output=True, text=True, env=denv, + timeout=600) + print(f"deleted {len(chunk)}: rc={rr.returncode} {rr.stdout[:200]}") + + if do_topup: + # Only non-CUA here; CUA refills go through cua_fill.py's throttle. + shortc = [] + for arm in ARMS: + for t in noncua: + c = cells.get(f"{arm}|{t}|terra") + held = sum(c[k] for k in ("VALID", "PENDING", "INFRA", "OTHER", + "SKIPPED")) if c else 0 + if held < N_NONCUA: + shortc.append((arm, t, N_NONCUA - held)) + if not shortc: + print(f"topup: nothing short — all non-CUA cells hold {N_NONCUA}") + for arm, t, gap in shortc: + print(f"topup: {arm}/{t} short by {gap}") + for attempt in range(1, 6): + rr = subprocess.run( + [ODDISH, "run", "-p", f"{HERE}/ds-{arm.lower()}", "-t", t, + "-a", "codex", "-m", "openai/gpt-5.6-terra", + "--n-trials", str(N_NONCUA), "-e", "modal", + "-E", EXP_OF[arm], "--override-memory-mb", "65536", + "--no-baseline-gate", + "--ak", f"reasoning_effort={arm.lower()}", + "--ae", f"ODDISH_EVAL_NONCE={arm}-{t}-{now}-{attempt}", + "--background", "--force"], + capture_output=True, text=True, env=env, timeout=400) + if "new version" in rr.stdout.lower(): + print(f" ABORT {arm}/{t}: submit wants a NEW TASK VERSION") + break + if "Task submitted!" in rr.stdout: + line = [l.strip() for l in rr.stdout.splitlines() if "Trials:" in l] + print(f" ok {line[0] if line else ''}") + break + print(f" attempt {attempt} failed") + if attempt < 5: + subprocess.run(["sleep", str(5 * attempt * attempt)]) + + +if __name__ == "__main__": + main() diff --git a/eval-ops/clear_broken.py b/eval-ops/clear_broken.py new file mode 100644 index 000000000..2096c6116 --- /dev/null +++ b/eval-ops/clear_broken.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +"""Delete the credential-failure trials in an experiment, keep the real ones. + +The opencode/google trials that predate the GOOGLE_GENERATIVE_AI_API_KEY mirror +never reached the model: `Command failed (exit 1)` on the opencode launch line, +zero tokens, no trajectory. They carry no signal and must not be scored. + +Trials that ran for real (hundreds of thousands of input tokens) are KEPT -- +including the post-fix smoke trials. + +A trial still in flight is never deleted: it may be a good one mid-run. + +Usage: clear_broken.py [--apply] +""" +import json, subprocess, sys, os, collections + +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +PENDING = {"pending", "queued", "running", "blocked", "preparing", "submitted", + "claimed", "in_progress", "initializing", "retrying"} +TASKS = { + "biofabric-rust-rewrite": "biofabric-rust-rewrite-897c0fc8", + "embedding-eval": "embedding-eval-be1dc228", + "excel-clone": "excel-clone-685bff89", + "find-network-alignments": "find-network-alignments-b2ecebb7", + "jax-pytorch-rewrite": "jax-pytorch-rewrite-93c38683", + "kubernetes-rust-rewrite": "kubernetes-rust-rewrite-bae0f616", + "mastodon-clone": "mastodon-clone-e9964b7a", + "nextjs-vite-rewrite": "nextjs-vite-rewrite-b0d028b0", + "parameter-golf": "parameter-golf-61e11605", + "post-train-ifeval-gpu": "post-train-ifeval-gpu-1de1166a", + "ruby-rust-port": "ruby-rust-port-383776ff", + "rust-c-compiler": "rust-c-compiler-57913cbe", + "rust-java-lsp": "rust-java-lsp-0a9c67d6", + "s3-clone": "s3-clone-23996371", + "slack-clone": "slack-clone-b0a98beb", + "stripe-clone": "stripe-clone-66061f4d", + "trimul-cuda": "trimul-cuda-fc34aaba", + "vliw-kernel-optimization": "vliw-kernel-optimization-e4d25121", + "wasm-simd": "wasm-simd-40e18127", + "zstd-decoder": "zstd-decoder-72bbfded", +} + + +def load_env(): + env = dict(os.environ) + for line in open("/home/user/oddish/.env"): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + env[k.strip()] = v.strip().strip('"').strip("'") + return env + + +def api(args, env, tries=8): + import time + for i in range(tries): + try: + r = subprocess.run([ODDISH] + args, capture_output=True, text=True, + env=env, timeout=600) + if r.returncode == 0 and r.stdout.strip(): + return json.loads(r.stdout) + except Exception: + pass + time.sleep(min(5 * (i + 1), 30)) + return None + + +def main(): + exp = sys.argv[1] + apply = "--apply" in sys.argv + env = load_env() + broken, kept, inflight, unread = [], 0, 0, [] + + for name, tid in TASKS.items(): + d = api(["status", tid, "--json"], env) + if d is None: + unread.append(name) + continue + for tr in d.get("trials") or []: + if tr.get("experiment_id") != exp: + continue + if tr.get("superseded_by_trial_id") is not None: + continue + st = (tr.get("status") or "").lower() + if st in PENDING: + inflight += 1 + continue + err = (tr.get("error_message") or "").lower() + tok = tr.get("input_tokens") or 0 + # Never reached the model: the launch failed, or it burned no tokens + # at all. Either way there is nothing to score. + if "command failed" in err or tok == 0: + broken.append((name, tr.get("id"), tok, (tr.get("error_message") or "")[:50])) + else: + kept += 1 + + if unread: + sys.exit(f"ABORT: {len(unread)} tasks unreadable ({unread[:5]}). " + "Refusing to delete on partial data.") + + print(f"broken (delete): {len(broken)} real (keep): {kept} in flight (skip): {inflight}") + for name, tid, tok, err in broken[:8]: + print(f" {name:26} {tid:34} tok={tok} {err}") + if len(broken) > 8: + print(f" ... and {len(broken)-8} more") + + if not apply: + print("(dry run — pass --apply to delete)") + return + if not broken: + return + denv = dict(env) + denv["ODDISH_API_KEY"] = denv["ODDISH_ADMIN_API_KEY"] + ids = [t[1] for t in broken] + for i in range(0, len(ids), 20): + chunk = ids[i:i + 20] + args = [ODDISH, "delete"] + sum([["-t", x] for x in chunk], []) + ["--json"] + r = subprocess.run(args, capture_output=True, text=True, env=denv, timeout=600) + print(f"deleted {len(chunk)}: rc={r.returncode}") + + +if __name__ == "__main__": + main() diff --git a/eval-ops/config.json b/eval-ops/config.json new file mode 100644 index 000000000..088bb0527 --- /dev/null +++ b/eval-ops/config.json @@ -0,0 +1,7 @@ +{ + "phase": "B", + "noncua_target": 10, + "cua_target": 10, + "cua_cap": 25, + "note": "Phase B. Charles 2026-08-06T03:0xZ: 'the concurrency can go up to 25. just queue up all of the trials for the cua. trust me.' -- cap raised 10 -> 25 on his explicit call, and the remaining CUA trials are queued outright to target rather than filled in waves. Oddish's own queue is the throttle now." +} \ No newline at end of file diff --git a/eval-ops/cua_dispatch.sh b/eval-ops/cua_dispatch.sh new file mode 100755 index 000000000..ad5eb8795 --- /dev/null +++ b/eval-ops/cua_dispatch.sh @@ -0,0 +1,50 @@ +#!/bin/bash +# Queue ALL remaining CUA trials straight to target, no wave-by-wave throttle. +# +# Charles, 2026-08-06: "the concurrency can go up to 25. just queue up all of +# the trials for the cua. trust me." The earlier <=10 wave filler existed +# because runbook 3.6 warns the browser grader hard-fails a trial (no reward +# file -> the trial errors) rather than queueing it when overloaded. That call +# is his; Oddish's own queue is the throttle from here. +# +# Target-based as always: --n-trials 10 creates 10 - existing per cell. +set -u +HERE=/home/user/terra-run +ODDISH=/home/user/oddish/oddish/.venv/bin/oddish +cd $HERE || exit 1 +set -a; source /home/user/oddish/.env; set +a +LOG=$HERE/cua_dispatch.log +N=$(python3 -c "import json;print(json.load(open('config.json'))['cua_target'])") + +TASKS="excel-clone mastodon-clone s3-clone slack-clone" +echo "=== CUA dispatch to N=$N start $(date -u +%FT%TZ) ===" >>$LOG + +for pair in "LOW 17b6f7d9 low" "MEDIUM c071229f medium" "HIGH a706e700 high"; do + set -- $pair + ARM=$1 EXP=$2 EFFORT=$3 + for T in $TASKS; do + ok=0 + for attempt in 1 2 3 4 5 6; do + out=$(cd $HERE/ds-${ARM,,} && timeout 400 $ODDISH run -p . -t "$T" \ + -a codex -m openai/gpt-5.6-terra --n-trials $N -e modal -E "$EXP" \ + --override-memory-mb 65536 --no-baseline-gate \ + --ak "reasoning_effort=$EFFORT" \ + --ae "ODDISH_EVAL_NONCE=$ARM-$T-$(date +%s%N)" \ + --background --force 2>&1) + if echo "$out" | grep -qi 'new version'; then + echo "$(date -u +%T) $ARM $T -> ABORT: wants NEW TASK VERSION" >>$LOG + break + fi + if echo "$out" | grep -q 'Task submitted!'; then + tr=$(echo "$out" | grep -Eio 'trials: *[0-9]+' | head -1) + echo "$(date -u +%T) $ARM $T attempt$attempt -> submitted ($tr)" >>$LOG + ok=1; break + fi + err=$(echo "$out" | grep -Eo 'HTTP [0-9]+|Internal Server Error' | head -1) + echo "$(date -u +%T) $ARM $T attempt$attempt -> FAIL ${err:-TIMEOUT/NO-OUTPUT}" >>$LOG + sleep $((5 * attempt * attempt)) + done + [ $ok -eq 1 ] || echo "$(date -u +%T) $ARM $T -> GAVE UP" >>$LOG + done +done +echo "=== CUA dispatch done $(date -u +%FT%TZ) ===" >>$LOG diff --git a/eval-ops/cua_fill.py b/eval-ops/cua_fill.py new file mode 100644 index 000000000..788ba5486 --- /dev/null +++ b/eval-ops/cua_fill.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +"""Throttled filler for the 4 CUA-verifier tasks. + +Runbook 3.6 caps concurrent CUA trials at ~10: the browser verifier runs on the +shared platform ANTHROPIC_API_KEY, and each CUA task.toml hard-fails (emits no +reward file, so the trial ERRORS instead of scoring) when the grader dies for an +infra reason. Over-parallelising does not just queue -- it destroys trials. +The cap is a verifier-safety limit, NOT a target: it does not relax in phase B. + +Each pass: count judge-consuming trials in flight, and submit only enough +1 +sweeps to refill up to the cap. Sweeps are target-based, so `--n-trials held+1` +adds exactly one trial to that cell. + +`nop` does NOT count against the cap: its verifier short-circuits in ~1.6 s +because the agent never brings the app up. + +Targets come from config.json so a phase flip needs no relaunch. + +Usage: cua_fill.py [--target N] [--cap N] [--dry] +""" +import json, subprocess, sys, os, datetime + +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +HERE = "/home/user/terra-run" +EXPS = {"17b6f7d9": "LOW", "c071229f": "MEDIUM", "a706e700": "HIGH"} +EXP_OF = {v: k for k, v in EXPS.items()} +CUA_TASKS = { + "excel-clone": "excel-clone-685bff89", + "mastodon-clone": "mastodon-clone-e9964b7a", + "s3-clone": "s3-clone-23996371", + "slack-clone": "slack-clone-b0a98beb", +} +# "retrying" is pending, not failed -- Oddish is re-running the trial itself. +# Omitting it understates in-flight (a retrying trial counts as neither valid +# nor pending) and so overstates headroom. Same fix as poll_all.py. +PENDING = ("pending", "queued", "running", "blocked", "preparing", "submitted", + "claimed", "in_progress", "initializing", "retrying") +OK_ERR = ("agenttimeouterror", "verifiertimeouterror", "timed out after") + + +def arg(flag, default): + return type(default)(sys.argv[sys.argv.index(flag) + 1]) if flag in sys.argv else default + + +def load_env(): + env = dict(os.environ) + for line in open("/home/user/oddish/.env"): + line = line.strip() + if line and not line.startswith("#") and "=" in line: + k, v = line.split("=", 1) + env[k.strip()] = v.strip().strip('"').strip("'") + return env + + +def scan(env): + cells, inflight, unknown = {}, 0, [] + for task, tid in CUA_TASKS.items(): + d = None + for _ in range(3): + r = subprocess.run([ODDISH, "status", tid, "--json"], + capture_output=True, text=True, env=env, timeout=600) + if r.returncode == 0 and r.stdout.strip(): + d = json.loads(r.stdout) + break + if d is None: + unknown.append(task) + continue + for tr in d.get("trials") or []: + arm = EXPS.get(tr.get("experiment_id")) + if arm is None or tr.get("superseded_by_trial_id") is not None: + continue + agent = tr.get("agent") or "?" + if agent in ("nop", "oracle"): + if (tr.get("status") or "").lower() in PENDING and agent == "oracle": + inflight += 1 + continue + c = cells.setdefault((arm, task), {"valid": 0, "pending": 0, "held": 0}) + c["held"] += 1 + st = (tr.get("status") or "").lower() + err = (tr.get("error_message") or "").lower() + if st in PENDING: + c["pending"] += 1 + inflight += 1 + elif not err or any(k in err for k in OK_ERR): + c["valid"] += 1 + return cells, inflight, unknown + + +def run(target, cap, dry): + env = load_env() + now = datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds") + cells, inflight, unknown = scan(env) + + grid = [] + for arm in ("LOW", "MEDIUM", "HIGH"): + for task in CUA_TASKS: + grid.append((arm, task, + cells.get((arm, task), {"valid": 0, "pending": 0, "held": 0}))) + + done = sum(1 for _, _, c in grid if c["held"] >= target) + tv = sum(c["valid"] for _, _, c in grid) + print(f"{now} CUA fill: target={target}/cell cap={cap} in-flight={inflight} " + f"valid={tv}/{12*target} cells_at_target={done}/12" + + (f" UNKNOWN={unknown}" if unknown else "")) + for arm, task, c in grid: + print(f" {arm:6} {task:15} valid={c['valid']} pending={c['pending']} held={c['held']}") + + if unknown: + print("ABORT: could not read every CUA task; refusing to submit blind " + "(an unread task could already be at the cap).") + return + + headroom = cap - inflight + need = sorted([(c["held"], arm, task, c) for arm, task, c in grid + if c["held"] < target]) + if not need: + print("all 12 cells hold their target — CUA fill complete") + return + if headroom <= 0: + print(f"at cap ({inflight}/{cap}) — nothing submitted this pass") + return + + sent = 0 + for held, arm, task, c in need: + if headroom <= 0: + break + n = c["held"] + 1 + print(f"submit {arm}/{task}: held={c['held']} -> --n-trials {n}") + if dry: + headroom -= 1 + sent += 1 + continue + for attempt in range(1, 6): + r = subprocess.run( + [ODDISH, "run", "-p", f"{HERE}/ds-{arm.lower()}", "-t", task, + "-a", "codex", "-m", "openai/gpt-5.6-terra", + "--n-trials", str(n), "-e", "modal", "-E", EXP_OF[arm], + "--override-memory-mb", "65536", "--no-baseline-gate", + "--ak", f"reasoning_effort={arm.lower()}", + "--ae", f"ODDISH_EVAL_NONCE={arm}-{task}-{now}-{attempt}", + "--background", "--force"], + capture_output=True, text=True, env=env, timeout=400) + if "new version" in r.stdout.lower(): + print(f" ABORT {arm}/{task}: submit wants a NEW TASK VERSION " + f"— dataset copy has drifted from the uploaded task.") + return + if "Task submitted!" in r.stdout: + line = [l.strip() for l in r.stdout.splitlines() if "Trials:" in l] + print(f" ok {line[0] if line else ''}") + headroom -= 1 + sent += 1 + break + print(f" attempt {attempt} failed") + if attempt < 5: + subprocess.run(["sleep", str(5 * attempt * attempt)]) + print(f"submitted {sent} this pass; in-flight now ~{inflight + sent}/{cap}") + + +def main(): + cfg = json.load(open(f"{HERE}/config.json")) + target = arg("--target", cfg["cua_target"]) + cap = arg("--cap", cfg["cua_cap"]) + dry = "--dry" in sys.argv + + # Single-writer lock. A pass routinely outlasts the babysit interval, and + # two concurrent fillers would each read a stale in-flight count and submit + # against the same headroom -- silently doubling us past the cap. + lock = f"{HERE}/.cua_fill.lock" + if not dry: + try: + fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY) + os.write(fd, str(os.getpid()).encode()) + os.close(fd) + except FileExistsError: + try: + pid = int(open(lock).read().strip()) + os.kill(pid, 0) + print(f"another cua_fill (pid {pid}) is running — skipping this pass") + return + except (ValueError, ProcessLookupError, PermissionError): + print("stale lock — taking over") + os.unlink(lock) + open(lock, "w").write(str(os.getpid())) + try: + run(target, cap, dry) + finally: + if not dry and os.path.exists(lock): + try: + if int(open(lock).read().strip()) == os.getpid(): + os.unlink(lock) + except ValueError: + pass + + +if __name__ == "__main__": + main() diff --git a/eval-ops/cua_loop.sh b/eval-ops/cua_loop.sh new file mode 100755 index 000000000..7b525909e --- /dev/null +++ b/eval-ops/cua_loop.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# Keep the CUA pipeline topped up to the cap without waiting for the 30-min +# babysit cron. CUA trials finish in ~15-30 min, so a half-hour refill interval +# leaves the pipeline running well under the cap for most of each cycle. +# +# A pass routinely takes ~30 min: submissions retry through HTTP 500s with +# backoff. Give each pass room to finish rather than truncating it mid-submit. +# +# cua_fill.py is cap-aware, config-driven and lock-protected, so this can +# safely overlap with the cron's own invocation -- whichever gets the lock does +# the work, and a phase flip is picked up without relaunching this loop. +set -u +cd /home/user/terra-run +for i in $(seq 1 200); do + out=$(timeout 5400 python3 -u cua_fill.py 2>&1) + echo "$(date -u +%FT%TZ) pass $i" + echo "$out" | grep -E 'CUA fill:|submitted [0-9]+ this pass|at cap|all 12 cells|ABORT|skipping' + if echo "$out" | grep -q 'all 12 cells hold their target'; then + echo "CUA FILL COMPLETE $(date -u +%FT%TZ)" + break + fi + sleep 120 +done diff --git a/eval-ops/cycle.sh b/eval-ops/cycle.sh new file mode 100755 index 000000000..ee3d60f9e --- /dev/null +++ b/eval-ops/cycle.sh @@ -0,0 +1,57 @@ +#!/bin/bash +# ONE self-healing pass of the phase-B fill. Safe to run from a cold container. +# +# Design note: this container is being recycled roughly every 45 minutes, which +# wipes /home/user/terra-run and kills any long-running loop. So the unit of +# work is a single idempotent pass that (a) rebuilds itself from git if needed +# and (b) re-asserts targets. Sweeps are target-based, so a pass that runs +# twice, or runs after a crash halfway through, costs nothing extra. +# +# Ordering is deliberate: canary first (probe the WRITE path -- during the +# 02:00Z degradation reads answered fine while every submit 500'd), then the +# non-CUA arms serially, then one throttled CUA pass. +set -u +HERE=/home/user/terra-run +ODDISH=/home/user/oddish/oddish/.venv/bin/oddish +LOG=$HERE/cycle.log + +# --- self-heal: restore from the git backup if the container was recycled +if [ ! -f $HERE/dispatch.sh ]; then + mkdir -p $HERE + cd /home/user/oddish || exit 1 + git fetch origin claude/oddish-api-env-setup-f76jpp >/dev/null 2>&1 + git merge --ff-only origin/claude/oddish-api-env-setup-f76jpp >/dev/null 2>&1 + cp /home/user/oddish/eval-ops/* $HERE/ 2>/dev/null + echo "$(date -u +%FT%TZ) RECOVERED toolkit from git" >>$LOG +fi +for a in low medium high; do + [ -d $HERE/ds-$a ] || cp -r /home/user/swe-marathon/tasks $HERE/ds-$a +done + +cd $HERE || exit 1 +set -a; source /home/user/oddish/.env; set +a +N=$(python3 -c "import json;print(json.load(open('config.json'))['noncua_target'])") + +# --- canary: re-assert a cell already at target. Creates ZERO trials, so it is +# a free probe of upload+submit -- the path that actually has to work. +out=$(cd ds-low && timeout 300 $ODDISH run -p . -t biofabric-rust-rewrite \ + -a codex -m openai/gpt-5.6-terra --n-trials $N -e modal -E 17b6f7d9 \ + --override-memory-mb 65536 --no-baseline-gate --ak reasoning_effort=low \ + --ae "ODDISH_EVAL_NONCE=canary-$(date +%s%N)" --background --force 2>&1) +if ! echo "$out" | grep -q 'Task submitted!'; then + echo "$(date -u +%FT%TZ) write path DOWN — skipping this pass" >>$LOG + exit 0 +fi +echo "$(date -u +%FT%TZ) write path UP — dispatching (target $N)" >>$LOG + +rm -f dispatch-LOW.log dispatch-MEDIUM.log dispatch-HIGH.log +bash dispatch.sh >/dev/null 2>&1 +created=$(grep -h submitted dispatch-*.log 2>/dev/null \ + | grep -oE 'Trials: *[0-9]+' | awk '{s+=$2} END {print s+0}') +ok=$(grep -h submitted dispatch-*.log 2>/dev/null | wc -l) +echo "$(date -u +%FT%TZ) non-CUA: ok=$ok/48 created=$created" >>$LOG + +timeout 3000 python3 -u cua_fill.py >>cua_loop.log 2>&1 +tail -1 cua_loop.log >>$LOG + +echo "$(date -u +%FT%TZ) pass done" >>$LOG diff --git a/eval-ops/dispatch.sh b/eval-ops/dispatch.sh new file mode 100755 index 000000000..568159f1b --- /dev/null +++ b/eval-ops/dispatch.sh @@ -0,0 +1,60 @@ +#!/bin/bash +# Raise the terra arm to N trials per task in all three effort experiments. +# N comes from config.json (phase B = 10). +# +# Sweeps are TARGET-based (runbook 3.1): --n-trials N creates N - existing for +# that (task, agent, model, experiment). Re-running is therefore safe and never +# double-submits, which matters because the API intermittently times out or +# 500s under load without telling us whether the submit committed. +# +# The 4 CUA-verifier tasks are deliberately NOT here -- they go through +# cua_fill.py, which throttles to the runbook 3.6 concurrency cap. +set -u +set -a; source /home/user/oddish/.env; set +a + +ODDISH=/home/user/oddish/oddish/.venv/bin/oddish +HERE=/home/user/terra-run +N=$(python3 -c "import json;print(json.load(open('$HERE/config.json'))['noncua_target'])") +TASKS="biofabric-rust-rewrite embedding-eval find-network-alignments \ +jax-pytorch-rewrite kubernetes-rust-rewrite nextjs-vite-rewrite parameter-golf \ +post-train-ifeval-gpu ruby-rust-port rust-c-compiler rust-java-lsp stripe-clone \ +trimul-cuda vliw-kernel-optimization wasm-simd zstd-decoder" + +arm_run() { # $1=arm label $2=exp id $3=reasoning_effort + local ARM=$1 EXP=$2 EFFORT=$3 + local LOG=$HERE/dispatch-${ARM}.log + cd "$HERE/ds-${ARM,,}" || return 1 + echo "=== $ARM ($EXP, effort=$EFFORT) target=$N start $(date -u +%FT%TZ) ===" >>"$LOG" + for T in $TASKS; do + local nonce="${ARM}-${T}-$(date +%s%N)" + local ok=0 + for attempt in 1 2 3 4 5; do + out=$(timeout 400 $ODDISH run -p . -t "$T" -a codex -m openai/gpt-5.6-terra \ + --n-trials $N -e modal -E "$EXP" --override-memory-mb 65536 \ + --no-baseline-gate --ak "reasoning_effort=$EFFORT" \ + --ae "ODDISH_EVAL_NONCE=$nonce" --background --force 2>&1) + if echo "$out" | grep -qi 'new version'; then + echo "$(date -u +%T) $ARM $T -> ABORT: wants NEW TASK VERSION" >>"$LOG" + break + fi + if echo "$out" | grep -q 'Task submitted!'; then + trials=$(echo "$out" | grep -Eio 'trials: *[0-9]+' | head -1) + echo "$(date -u +%T) $ARM $T attempt$attempt -> submitted ($trials)" >>"$LOG" + ok=1; break + fi + err=$(echo "$out" | grep -Eo 'HTTP [0-9]+|Internal Server Error' | head -1) + echo "$(date -u +%T) $ARM $T attempt$attempt -> FAIL ${err:-TIMEOUT/NO-OUTPUT}" >>"$LOG" + sleep $((5 * attempt * attempt)) + done + [ $ok -eq 1 ] || echo "$(date -u +%T) $ARM $T -> GAVE UP" >>"$LOG" + done + echo "=== $ARM done $(date -u +%FT%TZ) ===" >>"$LOG" +} + +# Serial by arm on purpose: concurrent submitters are the documented trigger +# for HTTP 500s (which commit nothing), and a failing submit returns fast, so +# parallelism buys little while the backend is unhappy. +arm_run LOW 17b6f7d9 low +arm_run MEDIUM c071229f medium +arm_run HIGH a706e700 high +echo "DISPATCH COMPLETE $(date -u +%FT%TZ)" >>$HERE/dispatch-ALL.log diff --git a/eval-ops/even_fill.py b/eval-ops/even_fill.py new file mode 100644 index 000000000..0b11ff803 --- /dev/null +++ b/eval-ops/even_fill.py @@ -0,0 +1,147 @@ +#!/usr/bin/env python3 +"""One even-fill pass: top every task up toward --target while respecting a +GLOBAL in-flight --cap, spreading new work across tasks instead of draining one. + +Why round-robin: a plain `--n-trials 10` sweep hands the whole backlog to the +scheduler at once, which then runs whatever it likes -- in practice one task at +a time. Filling fewest-held-first means every task advances together, so a +partial run is still a usable cross-section of the dataset. + +Sweeps are TARGET-based (runbook 3.1): `--n-trials N` creates N - existing for +that (task, agent, model, experiment). So submitting held+1 creates exactly one +trial, and re-running this pass is never double-submission. + +held = valid + pending. Failed/deleted trials do not count toward the target, +which is why a cell can sit below target with plenty of trials in its history. + +--ak flags are load-bearing: opencode's `variant` is a CliFlag with NO default, +so omitting `--ak variant=high` silently runs a different configuration than the +original arm. Pass every knob explicitly, every time. +""" +import argparse, json, subprocess, sys, datetime + +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +PENDING = ("pending", "queued", "running", "blocked", "preparing", "submitted", + "claimed", "in_progress", "initializing", "retrying") +OK_ERR = ("agenttimeouterror", "verifiertimeouterror", "timed out after") + + +def api(args, tries=6): + for _ in range(tries): + try: + r = subprocess.run([ODDISH] + args, capture_output=True, text=True, + timeout=600) + if r.returncode == 0 and r.stdout.strip(): + return json.loads(r.stdout) + except Exception: + pass + return None + + +def stamp(): + return datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds") + + +def main(): + p = argparse.ArgumentParser() + p.add_argument("exp") + p.add_argument("agent") + p.add_argument("model") + p.add_argument("--target", type=int, required=True) + p.add_argument("--cap", type=int, required=True, help="global in-flight cap") + p.add_argument("--batch", type=int, default=0, + help="max NEW trials this pass (0 = fill all headroom)") + p.add_argument("--ak", action="append", default=[], help="agent kwarg, repeatable") + p.add_argument("--dataset", default="/home/user/terra-run/ds-none") + p.add_argument("--dry-run", action="store_true") + a = p.parse_args() + + d = api(["status", a.exp, "--json"]) + if d is None: + sys.exit(f"{stamp()} experiment fetch FAILED -- refusing to submit blind") + tasks = {t["id"]: t["name"] for t in d.get("tasks") or []} + + cells, missing = {}, [] + for tid, tname in tasks.items(): + td = api(["status", tid, "--json"]) + if td is None: + missing.append(tname) + continue + held = pend = valid = 0 + for tr in td.get("trials") or []: + if tr.get("experiment_id") != a.exp or tr.get("superseded_by_trial_id"): + continue + st = (tr.get("status") or "").lower() + err = (tr.get("error_message") or "").lower() + if st in PENDING: + pend += 1 + held += 1 + elif not err or any(k in err for k in OK_ERR): + valid += 1 + held += 1 + cells[tname] = {"held": held, "pend": pend, "valid": valid} + + # A failed read looks exactly like an empty cell, and acting on that + # over-submits. Refuse the pass instead of guessing. + if missing: + sys.exit(f"{stamp()} {len(missing)} task(s) unreadable " + f"({', '.join(missing[:4])}) -- skipping this pass") + + in_flight = sum(c["pend"] for c in cells.values()) + headroom = a.cap - in_flight + # Batch size is the real throttle. Gemini's 20M input-tokens/minute quota is + # shared across every concurrent trial, so a big burst makes ALL of them 429 + # -- and a 429 makes opencode retry with the full context re-sent, which + # inflates token use further. Small batches keep the fleet under the ceiling. + if a.batch: + headroom = min(headroom, a.batch) + short = [(c["held"], n) for n, c in cells.items() if c["held"] < a.target] + print(f"{stamp()} in_flight={in_flight}/{a.cap} headroom={max(headroom,0)} " + f"short={len(short)} valid={sum(c['valid'] for c in cells.values())}" + f"/{a.target * len(cells)}") + + if not short: + print("COMPLETE -- every task at target") + return 0 + if headroom <= 0: + print("at cap -- nothing submitted") + return 1 + + created = 0 + for held, name in sorted(short): # fewest-held first == even progress + if headroom <= 0: + break + n = held + 1 # target-based: creates exactly one + cmd = [ODDISH, "run", "-p", a.dataset, "-t", name, "-a", a.agent, + "-m", a.model, "--n-trials", str(n), "-e", "modal", + "-E", a.exp, "--force", "--background"] + for kv in a.ak: + cmd += ["--ak", kv] + if a.dry_run: + print(f" DRY {name:<32} held={held} -> --n-trials {n}") + headroom -= 1 + continue + ok = False + for attempt in (1, 2, 3): + try: + r = subprocess.run(cmd, capture_output=True, text=True, timeout=900) + out = r.stdout + r.stderr + except Exception as e: + out = str(e) + if "Task submitted!" in out: + ok = True + break + if "new version" in out.lower() or "budget" in out.lower(): + print(f" {name:<32} ABORT: {out.strip().splitlines()[-1][:80]}") + break + subprocess.run(["sleep", str(10 * attempt)]) + print(f" {name:<32} held={held} -> n={n} {'ok' if ok else 'FAILED'}") + if ok: + created += 1 + headroom -= 1 + print(f"{stamp()} created={created}") + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/eval-ops/even_loop.sh b/eval-ops/even_loop.sh new file mode 100755 index 000000000..b950eed63 --- /dev/null +++ b/eval-ops/even_loop.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# Keep the opencode/gemini arm filling, evenly, in small batches, until every +# task holds 10 valid trials. +# +# Two throttles, both deliberate: +# --batch 3 queue three at a time, spread fewest-held-first, so all 20 tasks +# advance together instead of one task draining the whole quota. +# --cap 20 global in-flight ceiling. Measured, not guessed: at 34-35 +# concurrent every single completion came back 429 RESOURCE_EXHAUSTED +# (Gemini allows 20M input tokens/minute, shared fleet-wide), while +# at ~15 concurrent completions come back clean with real rewards. +# +# --ak variant=high is LOAD-BEARING. opencode's `variant` is a CliFlag with no +# default, so dropping it silently runs a different configuration than the arm +# we are trying to reproduce. +# +# Self-heals from git if the container is recycled mid-run. +set -u +HERE=/home/user/terra-run +LOG=$HERE/even_loop.log + +for pass in $(seq 1 4000); do + if [ ! -f $HERE/even_fill.py ]; then + cd /home/user/oddish || exit 1 + git fetch origin claude/oddish-api-env-setup-f76jpp >/dev/null 2>&1 + git merge --ff-only origin/claude/oddish-api-env-setup-f76jpp >/dev/null 2>&1 + mkdir -p $HERE && cp /home/user/oddish/eval-ops/* $HERE/ 2>/dev/null + echo "$(date -u +%FT%TZ) RECOVERED toolkit from git" >>$LOG + fi + [ -d $HERE/ds-none ] || cp -r /home/user/swe-marathon/tasks $HERE/ds-none + + cd $HERE || exit 1 + set -a; source /home/user/oddish/.env; set +a + + out=$(timeout 1500 python3 -u even_fill.py 826d7d88 opencode \ + google/gemini-3.7-flash --target 10 --cap 20 --batch 3 \ + --ak variant=high 2>&1) + echo "$out" | sed "s/^/[pass $pass] /" >>$LOG + + if echo "$out" | grep -q 'COMPLETE'; then + echo "$(date -u +%FT%TZ) OPENCODE FILL COMPLETE — 20 tasks x 10 valid" >>$LOG + break + fi + sleep 60 +done diff --git a/eval-ops/oc_poll.py b/eval-ops/oc_poll.py new file mode 100644 index 000000000..bb58c89ea --- /dev/null +++ b/eval-ops/oc_poll.py @@ -0,0 +1,67 @@ +#!/usr/bin/env python3 +"""Poll the opencode experiment: per-task counts + the newest completions. + +Classifies by error_message (runbook 3.5). The question this answers is +narrow: are trials that FINISH coming back clean, or still dying on 429s? +""" +import json, subprocess, sys, collections, datetime + +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +EXP = "826d7d88" +PENDING = ("pending","queued","running","blocked","preparing","submitted", + "claimed","in_progress","initializing","retrying") +OK_ERR = ("agenttimeouterror","verifiertimeouterror","timed out after") + +def api(a, tries=6): + for i in range(tries): + try: + r = subprocess.run([ODDISH]+a, capture_output=True, text=True, timeout=600) + if r.returncode == 0 and r.stdout.strip(): + return json.loads(r.stdout) + except Exception: + pass + return None + +d = api(["status", EXP, "--json"]) +if d is None: + sys.exit("experiment fetch FAILED") +tasks = {t["id"]: t["name"] for t in d.get("tasks") or []} + +rows, cells, miss = [], {}, [] +for tid, tname in tasks.items(): + td = api(["status", tid, "--json"]) + if td is None: + miss.append(tname); continue + c = {"valid":0,"pend":0,"infra":0,"pass":0} + for tr in td.get("trials") or []: + if tr.get("experiment_id") != EXP or tr.get("superseded_by_trial_id"): continue + st = (tr.get("status") or "").lower(); err = (tr.get("error_message") or "") + e = err.lower() + if st in PENDING: c["pend"] += 1; k = "PEND" + elif not e or any(x in e for x in OK_ERR): + c["valid"] += 1; k = "VALID" + if (tr.get("reward") or 0) and float(tr["reward"]) >= 1.0: c["pass"] += 1 + else: + c["infra"] += 1; k = "INFRA" + rows.append({"task":tname,"id":tr.get("id"),"k":k,"st":st, + "rew":tr.get("reward"),"in_tok":tr.get("input_tokens"), + "steps":tr.get("total_steps"),"cost":tr.get("cost_usd"), + "created":tr.get("created_at"),"fin":tr.get("finished_at") or tr.get("updated_at"), + "err":err[:110]}) + cells[tname] = c + +tot = collections.Counter() +for t, c in sorted(cells.items()): + for k in c: tot[k] += c[k] +print(f"{datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds')} " + f"valid={tot['valid']} pass={tot['pass']} pending={tot['pend']} infra={tot['infra']}" + + (f" MISSING={len(miss)}" if miss else "")) +for t, c in sorted(cells.items()): + print(f" {t:<34} valid={c['valid']:>2} pass={c['pass']:>2} pend={c['pend']:>2} infra={c['infra']:>3}") +fin = [r for r in rows if r["k"] != "PEND" and r["fin"]] +fin.sort(key=lambda r: r["fin"], reverse=True) +print("\nNEWEST 15 COMPLETIONS (most recent first)") +for r in fin[:15]: + print(f" {r['fin'][:19]} {str(r['id'])[-28:]:<30} {r['k']:<5} rew={r['rew']} " + f"in_tok={r['in_tok']} steps={r['steps']} {r['err']}") +json.dump({"cells":cells,"rows":rows}, open("/home/user/terra-run/oc_state.json","w"), indent=1) diff --git a/eval-ops/opencode_dispatch.sh b/eval-ops/opencode_dispatch.sh new file mode 100755 index 000000000..7dc4eb3ff --- /dev/null +++ b/eval-ops/opencode_dispatch.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# Re-run the opencode/gemini eval at 10 trials/task now that the credential +# mirror is deployed (verified: ~380k input tokens vs 0 before). +# Serial, one task per command, with backoff -- concurrent submits 500. +set -u +HERE=/home/user/terra-run +ODDISH=/home/user/oddish/oddish/.venv/bin/oddish +set -a; source /home/user/oddish/.env; set +a +LOG=$HERE/opencode-dispatch.log +cd $HERE/ds-none || exit 1 +TASKS="biofabric-rust-rewrite embedding-eval excel-clone find-network-alignments \ +jax-pytorch-rewrite kubernetes-rust-rewrite mastodon-clone nextjs-vite-rewrite \ +parameter-golf post-train-ifeval-gpu ruby-rust-port rust-c-compiler rust-java-lsp \ +s3-clone slack-clone stripe-clone trimul-cuda vliw-kernel-optimization wasm-simd \ +zstd-decoder" +echo "=== opencode rerun to 10 start $(date -u +%FT%TZ) ===" >>$LOG +for T in $TASKS; do + for attempt in 1 2 3 4 5; do + out=$(timeout 900 $ODDISH run -p . -t "$T" -a opencode -m google/gemini-3.7-flash \ + --n-trials 10 -e modal -E 826d7d88 --force --background 2>&1) + if echo "$out" | grep -qi 'new version'; then + echo "$(date -u +%T) $T -> ABORT: NEW TASK VERSION" >>$LOG; break; fi + if echo "$out" | grep -qi 'budget'; then + echo "$(date -u +%T) $T -> BUDGET LIMIT" >>$LOG; break; fi + if echo "$out" | grep -q 'Task submitted!'; then + echo "$(date -u +%T) $T attempt$attempt -> $(echo "$out" | grep -Eio 'trials: *[0-9]+' | head -1)" >>$LOG + break; fi + echo "$(date -u +%T) $T attempt$attempt -> FAIL" >>$LOG + sleep $((10 * attempt)) + done +done +echo "=== opencode rerun done $(date -u +%FT%TZ) ===" >>$LOG diff --git a/eval-ops/passk.py b/eval-ops/passk.py new file mode 100644 index 000000000..4566fa29f --- /dev/null +++ b/eval-ops/passk.py @@ -0,0 +1,171 @@ +#!/usr/bin/env python3 +"""pass@1 / pass@3 / pass@8, following the oddish runbook's rules. + +VALIDITY (docs/swe-marathon-eval-runbook.md, "Target outcome" + gotcha 3.5): + + valid = the trial ran to a real terminal state -- a clean success *or* an + AgentTimeoutError. Any other infra failure (non-zero agent exit like + 137/143/1, verifier infra failure, a Harbor ExceptionGroup) is NOT valid; + delete and rerun those. + + Classify by error_message, never job status: infra failures frequently + report status=success, reward=0 with a populated error_message. + +So infra trials are excluded from the denominator entirely -- they are failures +of the harness, not attempts by the model, and the runbook's remedy is to delete +and rerun them rather than score them. This differs from the dashboard, which +counts every completed attempt; both are printed so the gap is visible. + +ESTIMATOR (frontend/src/lib/pass-at-k.ts, the only definition in the repo): + + pass@k = 1 - C(n-c, k)/C(n, k), product form; n is PER TASK per agent; + c counts reward == 1; the reported value is the unweighted MEAN of the + per-task pass@k. nop/oracle baselines are excluded. + +Raw trials are cached to passk-raw.json so the numbers can be recomputed under +different rules without refetching. +""" +import json, subprocess, sys, os, collections + +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +RAW = "/home/user/terra-run/passk-raw.json" +KS = (1, 3, 8) +BASELINES = {"nop", "oracle"} +PENDING = {"pending", "queued", "running", "blocked", "preparing", "submitted", + "claimed", "in_progress", "initializing", "retrying"} +# Prose form matters: this fleet writes "Agent execution timed out after N +# seconds", not the exception-class name. Match both or a genuine timeout -- +# which the runbook counts as VALID -- gets thrown out as infra. +OK_ERR = ("agenttimeouterror", "verifiertimeouterror", "timed out after") + + +def api(args, tries=3): + for _ in range(tries): + try: + r = subprocess.run([ODDISH] + args, capture_output=True, text=True, + timeout=600) + if r.returncode == 0 and r.stdout.strip(): + return json.loads(r.stdout) + except Exception: + pass + return None + + +def fetch(exps): + raw = json.load(open(RAW)) if os.path.exists(RAW) else {} + for exp in exps: + if exp in raw: + print(f"{exp}: cached ({len(raw[exp]['trials'])} trials)", file=sys.stderr) + continue + d = api(["status", exp, "--json"]) + if d is None: + print(f"{exp}: FETCH FAILED", file=sys.stderr) + continue + tasks = {t["id"]: t["name"] for t in d.get("tasks") or []} + name = next((t.get("experiment_name") for t in (d.get("tasks") or []) + if t.get("experiment_id") == exp), exp) + trials = [] + for tid, tname in tasks.items(): + td = api(["status", tid, "--json"]) + if td is None: + print(f" {exp}/{tname}: FETCH FAILED", file=sys.stderr) + continue + for tr in td.get("trials") or []: + if tr.get("experiment_id") != exp: + continue + if tr.get("superseded_by_trial_id") is not None: + continue + trials.append({ + "task": tname, "agent": tr.get("agent") or "?", + "model": tr.get("model") or "?", + "status": (tr.get("status") or "").lower(), + "reward": tr.get("reward"), + "err": (tr.get("error_message") or "")}) + raw[exp] = {"name": name, "trials": trials} + print(f"{exp} ({name}): {len(trials)} trials over {len(tasks)} tasks", + file=sys.stderr) + with open(RAW, "w") as f: + json.dump(raw, f) + return raw + + +def pass_at_k(n, c, k): + """Verbatim port of calculatePassAtK (frontend/src/lib/pass-at-k.ts).""" + if k > n: + return 1.0 if c > 0 else 0.0 + if c == 0: + return 0.0 + if c >= n: + return 1.0 + if k <= 0: + return 0.0 + if k > n - c: + return 1.0 + p = 1.0 + for i in range(k): + p *= (n - c - i) / (n - i) + return 1.0 - p + + +def is_valid(tr): + """Runbook validity: clean terminal state, or an honest agent/verifier + timeout. Everything else with an error_message is infra.""" + err = tr["err"].lower() + if not err: + return True + return any(k in err for k in OK_ERR) + + +def curve(raw, exps, rule): + """rule='runbook' -> valid trials only; rule='dashboard' -> all completed.""" + rows = [] + for exp in exps: + if exp not in raw: + continue + per = collections.defaultdict(lambda: collections.defaultdict( + lambda: {"n": 0, "c": 0})) + infra = collections.Counter() + for tr in raw[exp]["trials"]: + if tr["agent"] in BASELINES or tr["status"] in PENDING: + continue + key = f"{tr['agent']} / {tr['model']}" + if rule == "runbook" and not is_valid(tr): + infra[key] += 1 + continue + cell = per[key][tr["task"]] + cell["n"] += 1 + if tr["reward"] is not None and float(tr["reward"]) == 1.0: + cell["c"] += 1 + for key, tasks in per.items(): + vals = [sum(pass_at_k(v["n"], v["c"], k) for v in tasks.values()) + / len(tasks) for k in KS] + rows.append({ + "exp": exp, "name": raw[exp]["name"], "key": key, + "tasks": len(tasks), "trials": sum(v["n"] for v in tasks.values()), + "passes": sum(v["c"] for v in tasks.values()), + "infra_excluded": infra[key], + **{f"pass@{k}": v for k, v in zip(KS, vals)}}) + return rows + + +def show(title, rows): + print(f"\n{title}") + print("-" * 104) + print(f"{'experiment':<32}{'agent / model':<30}{'tasks':>6}{'trials':>7}" + f"{'pass':>6}" + "".join(f"{'pass@'+str(k):>9}" for k in KS) + + f"{'infra':>7}") + print("-" * 104) + for r in rows: + print(f"{r['name'][:30]:<32}{r['key'][:28]:<30}{r['tasks']:>6}" + f"{r['trials']:>7}{r['passes']:>6}" + + "".join(f"{100*r['pass@'+str(k)]:>8.1f}%" for k in KS) + + f"{r['infra_excluded']:>7}") + + +if __name__ == "__main__": + exps = sys.argv[1:] + raw = fetch(exps) + rb = curve(raw, exps, "runbook") + show("RUNBOOK RULE — valid trials only (infra deleted/rerun, so not scored)", rb) + show("DASHBOARD RULE — every completed attempt counts", curve(raw, exps, "dashboard")) + json.dump(rb, open("/home/user/terra-run/passk.json", "w"), indent=1) diff --git a/eval-ops/poll_all.py b/eval-ops/poll_all.py new file mode 100644 index 000000000..347f11103 --- /dev/null +++ b/eval-ops/poll_all.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +"""Fleet poller across the three gpt-5.6-terra reasoning-effort experiments. + +Task IDs are content-derived and SHARED across experiments, so each task is +fetched once and its trials are bucketed by experiment_id. + +Classifies by error_message, never by job status (runbook 3.5). +Writes /home/user/terra-run/state.json for the babysit loop. +""" +import json, subprocess, datetime, collections, sys, os + +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +STATE = "/home/user/terra-run/state.json" + +EXPS = {"17b6f7d9": "LOW", "c071229f": "MEDIUM", "a706e700": "HIGH"} + +CUA_TASKS = { + "excel-clone": "excel-clone-685bff89", + "mastodon-clone": "mastodon-clone-e9964b7a", + "s3-clone": "s3-clone-23996371", + "slack-clone": "slack-clone-b0a98beb", +} +NONCUA_TASKS = { + "biofabric-rust-rewrite": "biofabric-rust-rewrite-897c0fc8", + "embedding-eval": "embedding-eval-be1dc228", + "find-network-alignments": "find-network-alignments-b2ecebb7", + "jax-pytorch-rewrite": "jax-pytorch-rewrite-93c38683", + "kubernetes-rust-rewrite": "kubernetes-rust-rewrite-bae0f616", + "nextjs-vite-rewrite": "nextjs-vite-rewrite-b0d028b0", + "parameter-golf": "parameter-golf-61e11605", + "post-train-ifeval-gpu": "post-train-ifeval-gpu-1de1166a", + "ruby-rust-port": "ruby-rust-port-383776ff", + "rust-c-compiler": "rust-c-compiler-57913cbe", + "rust-java-lsp": "rust-java-lsp-0a9c67d6", + "stripe-clone": "stripe-clone-66061f4d", + "trimul-cuda": "trimul-cuda-fc34aaba", + "vliw-kernel-optimization": "vliw-kernel-optimization-e4d25121", + "wasm-simd": "wasm-simd-40e18127", + "zstd-decoder": "zstd-decoder-72bbfded", +} +ALL_TASKS = {**NONCUA_TASKS, **CUA_TASKS} + +# VALID = reached a real terminal state: a clean run, or an honest agent/verifier +# timeout. This fleet spells timeouts in prose ("Agent execution timed out after +# N seconds"), not as the exception-class name -- match both or a real timeout +# gets misfiled as infra and deleted. +OK_ERR = ("agenttimeouterror", "verifiertimeouterror", "timed out after") +# "worker heartbeat stalled" is recoverable while status is "retrying" (Oddish +# re-runs it), but once it lands in a terminal status the trial is genuinely +# lost and must be deleted + re-run like any other infra failure. +INFRA_ERR = ("command failed", "exceptiongroup", "no reward file", + "ratelimit", "rate limit", "429", "internal server error", + "worker heartbeat stalled") +# "retrying" is a PENDING state, not a failure: Oddish is re-running the trial +# itself. It carries an error_message ("Worker heartbeat stalled for over 15 +# minutes"), so leaving it out of this list would classify it OTHER and +# --delete would destroy trials that were about to succeed on their own. +PENDING = ("pending", "queued", "running", "blocked", "preparing", "submitted", + "claimed", "in_progress", "initializing", "retrying") + + +def classify(tr): + st = (tr.get("status") or "").lower() + err = (tr.get("error_message") or "").lower() + if st in PENDING: + return "PENDING" + if st == "skipped": + return "SKIPPED" + if not err: + return "VALID" + if any(k in err for k in OK_ERR): + return "VALID" + if any(k in err for k in INFRA_ERR): + return "INFRA" + return "OTHER" + + +def fetch(task_id, tries=3): + for _ in range(tries): + try: + r = subprocess.run([ODDISH, "status", task_id, "--json"], + capture_output=True, text=True, timeout=600) + if r.returncode == 0 and r.stdout.strip(): + return json.loads(r.stdout) + except Exception: + pass + return None + + +def main(): + only = sys.argv[1:] or list(ALL_TASKS) + now = datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds") + state = {"polled_at": now, "cells": {}, "unknown": [], "trials": {}} + + for task in only: + d = fetch(ALL_TASKS[task]) + if d is None: + state["unknown"].append(task) + print(f"UNKNOWN (fetch failed) {task}", file=sys.stderr) + continue + for tr in d.get("trials") or []: + arm = EXPS.get(tr.get("experiment_id")) + if arm is None or tr.get("superseded_by_trial_id") is not None: + continue + agent = tr.get("agent") or "?" + kind = "terra" if agent not in ("nop", "oracle") else agent + k = classify(tr) + c = state["cells"].setdefault( + f"{arm}|{task}|{kind}", + {"VALID": 0, "PENDING": 0, "INFRA": 0, "OTHER": 0, + "SKIPPED": 0, "pass": 0}) + c[k] += 1 + if k == "VALID" and (tr.get("reward") or 0) and float(tr["reward"]) >= 1.0: + c["pass"] += 1 + state["trials"].setdefault(k, []).append({ + "arm": arm, "task": task, "agent": agent, "kind": kind, + "id": tr.get("id"), "status": tr.get("status"), + "reward": tr.get("reward"), "steps": tr.get("total_steps"), + "cost": tr.get("cost_usd"), + "err": (tr.get("error_message") or "")[:160]}) + + tmp = STATE + ".tmp" + with open(tmp, "w") as f: + json.dump(state, f, indent=1) + os.replace(tmp, STATE) + + tot = collections.Counter() + cua_live = 0 + for key, c in state["cells"].items(): + _, task, kind = key.split("|") + for k in ("VALID", "PENDING", "INFRA", "OTHER", "SKIPPED"): + tot[k] += c[k] + if task in CUA_TASKS and kind != "nop": + cua_live += c["PENDING"] + + print(f"{now} valid={tot['VALID']} pending={tot['PENDING']} " + f"infra={tot['INFRA']} other={tot['OTHER']} skipped={tot['SKIPPED']}") + print(f"CUA judge-consuming in-flight (oracle+terra, excl nop) = {cua_live} / cap 10") + if state["unknown"]: + print("UNKNOWN tasks:", ", ".join(state["unknown"])) + + +if __name__ == "__main__": + main() diff --git a/eval-ops/run_forever.sh b/eval-ops/run_forever.sh new file mode 100755 index 000000000..81dc168c6 --- /dev/null +++ b/eval-ops/run_forever.sh @@ -0,0 +1,63 @@ +#!/bin/bash +# Long-running driver for phase B. Runs until every cell holds its target. +# +# Long loops are viable again: the container was being reclaimed for INACTIVITY +# (three wipes), and a 20-minute session heartbeat now keeps it warm. The +# self-heal step stays anyway -- it is cheap and makes a cold start harmless. +# +# Each cycle: canary the write path, dispatch non-CUA, re-assert CUA. All +# target-based, so any cycle is a no-op where the work is already done. +set -u +HERE=/home/user/terra-run +ODDISH=/home/user/oddish/oddish/.venv/bin/oddish +LOG=$HERE/forever.log + +for cycle in $(seq 1 500); do + if [ ! -f $HERE/dispatch.sh ]; then + cd /home/user/oddish || exit 1 + git fetch origin claude/oddish-api-env-setup-f76jpp >/dev/null 2>&1 + git merge --ff-only origin/claude/oddish-api-env-setup-f76jpp >/dev/null 2>&1 + mkdir -p $HERE && cp /home/user/oddish/eval-ops/* $HERE/ 2>/dev/null + echo "$(date -u +%FT%TZ) RECOVERED from git" >>$LOG + fi + for a in low medium high; do + [ -d $HERE/ds-$a ] || cp -r /home/user/swe-marathon/tasks $HERE/ds-$a + done + + cd $HERE || exit 1 + set -a; source /home/user/oddish/.env; set +a + N=$(python3 -c "import json;print(json.load(open('config.json'))['noncua_target'])") + + # Canary: re-asserting a cell already at target creates ZERO trials, so this + # probes upload+submit for free. Reads can answer fine while writes 500. + out=$(cd ds-low && timeout 300 $ODDISH run -p . -t biofabric-rust-rewrite \ + -a codex -m openai/gpt-5.6-terra --n-trials $N -e modal -E 17b6f7d9 \ + --override-memory-mb 65536 --no-baseline-gate --ak reasoning_effort=low \ + --ae "ODDISH_EVAL_NONCE=canary-$(date +%s%N)" --background --force 2>&1) + if ! echo "$out" | grep -q 'Task submitted!'; then + echo "$(date -u +%FT%TZ) cycle $cycle: write path DOWN — backing off" >>$LOG + sleep 600 + continue + fi + + rm -f dispatch-LOW.log dispatch-MEDIUM.log dispatch-HIGH.log + bash dispatch.sh >/dev/null 2>&1 + nc_created=$(grep -h submitted dispatch-*.log 2>/dev/null \ + | grep -oE 'Trials: *[0-9]+' | awk '{s+=$2} END {print s+0}') + nc_ok=$(grep -h submitted dispatch-*.log 2>/dev/null | wc -l) + + : > cua_dispatch.log + bash cua_dispatch.sh >/dev/null 2>&1 + cua_created=$(grep -h submitted cua_dispatch.log 2>/dev/null \ + | grep -oE 'Trials: *[0-9]+' | awk '{s+=$2} END {print s+0}') + cua_ok=$(grep -h submitted cua_dispatch.log 2>/dev/null | wc -l) + + echo "$(date -u +%FT%TZ) cycle $cycle: nonCUA ok=$nc_ok/48 created=$nc_created | CUA ok=$cua_ok/12 created=$cua_created" >>$LOG + + if [ "$nc_ok" -ge 48 ] && [ "$nc_created" -eq 0 ] \ + && [ "$cua_ok" -ge 12 ] && [ "$cua_created" -eq 0 ]; then + echo "$(date -u +%FT%TZ) PHASE B COMPLETE — all 60 cells at target" >>$LOG + break + fi + sleep 600 +done From fa2a05572672a7ba20e7a4e78a70699c03094076 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 01:49:05 +0000 Subject: [PATCH 2/5] Count burnt trials when resolving the fill target The filler asked the server for `held + 1` trials, where held = valid + pending. But the server resolves --n-trials N by subtracting every trial it still holds for the cell, and it counts by status -- and infra failures report status=success with reward=0 and a populated error_message. Every terminal trial in the opencode arm has status=success, the 30 burnt ones included. So in any cell with a past failure, `held + 1` was less than or equal to what the server already had, and the submit created nothing while still printing "Task submitted!". excel-clone sat at held=0 with one burnt trial and got re-submitted on three consecutive passes, creating zero trials each time; the cells with eight burnt trials would never have filled at all. Track `existing` (everything the server still has) separately from `held` (progress toward the target), submit existing+1, and parse the trial count back out of the response so a divergence between the two notions is visible in the log rather than silent. --- eval-ops/even_fill.py | 52 ++++++++++++++++++++++++++++++++++--------- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/eval-ops/even_fill.py b/eval-ops/even_fill.py index 0b11ff803..5e211d784 100644 --- a/eval-ops/even_fill.py +++ b/eval-ops/even_fill.py @@ -8,17 +8,28 @@ partial run is still a usable cross-section of the dataset. Sweeps are TARGET-based (runbook 3.1): `--n-trials N` creates N - existing for -that (task, agent, model, experiment). So submitting held+1 creates exactly one -trial, and re-running this pass is never double-submission. +that (task, agent, model, experiment), so re-running a pass is never a double +submit. -held = valid + pending. Failed/deleted trials do not count toward the target, -which is why a cell can sit below target with plenty of trials in its history. +Two different counts matter, and conflating them stalls the fill silently: + + existing -- every trial the SERVER still has for the cell. It counts by + status, and infra failures report status=success with reward=0 + and a populated error_message, so a burnt trial keeps occupying a + slot. Submit `existing + 1` to create exactly one new trial. + held -- valid + pending, i.e. actual progress toward the target. Burnt + trials do not count here, which is why a cell can sit below + target with a long trial history. + +Asking for held+1 in a cell that has any past failure resolves to zero new +trials while still printing "Task submitted!", so the cell never fills and the +loop reports success forever. --ak flags are load-bearing: opencode's `variant` is a CliFlag with NO default, so omitting `--ak variant=high` silently runs a different configuration than the original arm. Pass every knob explicitly, every time. """ -import argparse, json, subprocess, sys, datetime +import argparse, json, re, subprocess, sys, datetime ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" PENDING = ("pending", "queued", "running", "blocked", "preparing", "submitted", @@ -67,10 +78,15 @@ def main(): if td is None: missing.append(tname) continue - held = pend = valid = 0 + held = pend = valid = existing = 0 for tr in td.get("trials") or []: if tr.get("experiment_id") != a.exp or tr.get("superseded_by_trial_id"): continue + # `existing` is what the SERVER counts when resolving --n-trials N. + # It counts by status, and infra failures report status=success with + # reward=0 and a populated error_message -- so a burnt trial still + # occupies a slot against the target even though it scores nothing. + existing += 1 st = (tr.get("status") or "").lower() err = (tr.get("error_message") or "").lower() if st in PENDING: @@ -79,7 +95,8 @@ def main(): elif not err or any(k in err for k in OK_ERR): valid += 1 held += 1 - cells[tname] = {"held": held, "pend": pend, "valid": valid} + cells[tname] = {"held": held, "pend": pend, "valid": valid, + "existing": existing} # A failed read looks exactly like an empty cell, and acting on that # over-submits. Refuse the pass instead of guessing. @@ -111,17 +128,22 @@ def main(): for held, name in sorted(short): # fewest-held first == even progress if headroom <= 0: break - n = held + 1 # target-based: creates exactly one + # Ask for existing+1, NOT held+1. The server subtracts what it already + # has, and it counts burnt trials too -- so held+1 resolves to zero new + # trials in any cell with a past failure, and that cell silently never + # fills while the loop reports "ok" forever. + n = cells[name]["existing"] + 1 # target-based: creates exactly one cmd = [ODDISH, "run", "-p", a.dataset, "-t", name, "-a", a.agent, "-m", a.model, "--n-trials", str(n), "-e", "modal", "-E", a.exp, "--force", "--background"] for kv in a.ak: cmd += ["--ak", kv] if a.dry_run: - print(f" DRY {name:<32} held={held} -> --n-trials {n}") + print(f" DRY {name:<32} held={held} existing={cells[name]['existing']}" + f" -> --n-trials {n}") headroom -= 1 continue - ok = False + ok, made = False, None for attempt in (1, 2, 3): try: r = subprocess.run(cmd, capture_output=True, text=True, timeout=900) @@ -130,12 +152,20 @@ def main(): out = str(e) if "Task submitted!" in out: ok = True + m = re.search(r"trials:\s*(\d+)", out, re.I) + made = int(m.group(1)) if m else None break if "new version" in out.lower() or "budget" in out.lower(): print(f" {name:<32} ABORT: {out.strip().splitlines()[-1][:80]}") break subprocess.run(["sleep", str(10 * attempt)]) - print(f" {name:<32} held={held} -> n={n} {'ok' if ok else 'FAILED'}") + # `made` is the server's own count of trials created. It should be 1; + # anything else means the server's notion of "existing" diverges from + # ours and the batch discipline is not holding. Surface it loudly. + flag = "" if made in (None, 1) else f" <-- CREATED {made}, EXPECTED 1" + print(f" {name:<32} held={held} existing={cells[name]['existing']}" + f" -> n={n} {'ok' if ok else 'FAILED'}" + + (f" made={made}" if made is not None else "") + flag) if ok: created += 1 headroom -= 1 From bcde797dbf0f74a73ea4a026608967aa54eef1b2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 04:03:52 +0000 Subject: [PATCH 3/5] Add a per-task in-flight mode to the filler Round-robin with a global cap keeps every task advancing, but it leaves tasks idle whenever the cap binds -- four consecutive passes submitted nothing at all because in-flight was already at the ceiling, so nothing progressed. --per-task-inflight N instead keeps N trials running for every task and lets Oddish's own concurrency control decide what actually executes. The per-task count is clamped to what the target still needs, so a cell close to its target does not overshoot on the last top-up. The created-count check earns its keep here: topping ruby-rust-port up by 3 created 10, meaning the server's notion of "existing" for that cell did not match ours. The pass flags any such divergence inline rather than reporting a uniform success. Also raises the per-pass timeout: a per-task pass polls 20 tasks and then issues up to 20 submits, each uploading the dataset, which does not fit in the window sized for a three-submit round-robin pass. --- eval-ops/even_fill.py | 66 +++++++++++++++++++++++++++++++++++++++++++ eval-ops/even_loop.sh | 29 +++++++++++-------- 2 files changed, 84 insertions(+), 11 deletions(-) diff --git a/eval-ops/even_fill.py b/eval-ops/even_fill.py index 5e211d784..ee4df14bd 100644 --- a/eval-ops/even_fill.py +++ b/eval-ops/even_fill.py @@ -62,6 +62,9 @@ def main(): p.add_argument("--cap", type=int, required=True, help="global in-flight cap") p.add_argument("--batch", type=int, default=0, help="max NEW trials this pass (0 = fill all headroom)") + p.add_argument("--per-task-inflight", type=int, default=0, + help="keep this many trials in flight for EVERY task; " + "overrides --batch/--cap round-robin mode") p.add_argument("--ak", action="append", default=[], help="agent kwarg, repeatable") p.add_argument("--dataset", default="/home/user/terra-run/ds-none") p.add_argument("--dry-run", action="store_true") @@ -105,6 +108,69 @@ def main(): f"({', '.join(missing[:4])}) -- skipping this pass") in_flight = sum(c["pend"] for c in cells.values()) + + # --- per-task in-flight mode ------------------------------------------- + # Keep N trials running for EVERY task, letting Oddish's own concurrency + # control decide what actually executes rather than throttling here. + if a.per_task_inflight: + want = {} + for name, c in cells.items(): + if c["held"] >= a.target: + continue + # Never queue past what the target still needs: held already counts + # the in-flight ones, so this is the remaining room, not the deficit. + n_new = min(a.per_task_inflight - c["pend"], a.target - c["held"]) + if n_new > 0: + want[name] = n_new + print(f"{stamp()} in_flight={in_flight} per_task_target={a.per_task_inflight} " + f"topping_up={len(want)} tasks (+{sum(want.values())}) " + f"valid={sum(c['valid'] for c in cells.values())}" + f"/{a.target * len(cells)}") + if not want: + if all(c["held"] >= a.target for c in cells.values()): + print("COMPLETE -- every task at target") + return 0 + print("every task already at its in-flight target") + return 1 + created = 0 + for name, n_new in sorted(want.items(), key=lambda kv: -kv[1]): + n = cells[name]["existing"] + n_new + cmd = [ODDISH, "run", "-p", a.dataset, "-t", name, "-a", a.agent, + "-m", a.model, "--n-trials", str(n), "-e", "modal", + "-E", a.exp, "--force", "--background"] + for kv in a.ak: + cmd += ["--ak", kv] + if a.dry_run: + print(f" DRY {name:<32} pend={cells[name]['pend']} " + f"existing={cells[name]['existing']} -> --n-trials {n} (+{n_new})") + continue + ok, made = False, None + for attempt in (1, 2, 3): + try: + r = subprocess.run(cmd, capture_output=True, text=True, timeout=900) + out = r.stdout + r.stderr + except Exception as e: + out = str(e) + if "Task submitted!" in out: + ok = True + m = re.search(r"trials:\s*(\d+)", out, re.I) + made = int(m.group(1)) if m else None + break + if "new version" in out.lower() or "budget" in out.lower(): + print(f" {name:<32} ABORT: {out.strip().splitlines()[-1][:80]}") + break + subprocess.run(["sleep", str(10 * attempt)]) + flag = "" if made in (None, n_new) else f" <-- CREATED {made}, EXPECTED {n_new}" + print(f" {name:<32} pend={cells[name]['pend']} " + f"existing={cells[name]['existing']} -> n={n} (+{n_new}) " + f"{'ok' if ok else 'FAILED'}" + + (f" made={made}" if made is not None else "") + flag) + if ok: + created += made if made is not None else n_new + print(f"{stamp()} created={created}") + return 1 + # --- end per-task mode -------------------------------------------------- + headroom = a.cap - in_flight # Batch size is the real throttle. Gemini's 20M input-tokens/minute quota is # shared across every concurrent trial, so a big burst makes ALL of them 429 diff --git a/eval-ops/even_loop.sh b/eval-ops/even_loop.sh index b950eed63..c8fd01ead 100755 --- a/eval-ops/even_loop.sh +++ b/eval-ops/even_loop.sh @@ -1,14 +1,17 @@ #!/bin/bash -# Keep the opencode/gemini arm filling, evenly, in small batches, until every -# task holds 10 valid trials. +# Keep the opencode/gemini arm filling, evenly, until every task holds 10 valid +# trials. # -# Two throttles, both deliberate: -# --batch 3 queue three at a time, spread fewest-held-first, so all 20 tasks -# advance together instead of one task draining the whole quota. -# --cap 20 global in-flight ceiling. Measured, not guessed: at 34-35 -# concurrent every single completion came back 429 RESOURCE_EXHAUSTED -# (Gemini allows 20M input tokens/minute, shared fleet-wide), while -# at ~15 concurrent completions come back clean with real rewards. +# --per-task-inflight 3 keep three trials running for EVERY task, so all 20 +# advance together instead of one draining the queue. +# Oddish has its own concurrency control, so this does +# not throttle the fleet further -- it just makes sure +# no task sits idle waiting its turn. +# +# Note on rate limits: Gemini allows 20M input tokens/minute shared fleet-wide. +# An earlier run at 34-35 concurrent had every completion come back +# 429 RESOURCE_EXHAUSTED. Watch the valid count -- if it stops climbing while +# trials keep finishing, the fleet is back over the ceiling. # # --ak variant=high is LOAD-BEARING. opencode's `variant` is a CliFlag with no # default, so dropping it silently runs a different configuration than the arm @@ -32,8 +35,12 @@ for pass in $(seq 1 4000); do cd $HERE || exit 1 set -a; source /home/user/oddish/.env; set +a - out=$(timeout 1500 python3 -u even_fill.py 826d7d88 opencode \ - google/gemini-3.7-flash --target 10 --cap 20 --batch 3 \ + # A per-task pass polls 20 tasks and then issues up to 20 submits, each of + # which uploads the dataset -- far longer than the round-robin pass this + # timeout was sized for. A kill mid-pass is harmless (target-based, the next + # pass resumes) but wastes the poll, so give it room to finish. + out=$(timeout 3000 python3 -u even_fill.py 826d7d88 opencode \ + google/gemini-3.7-flash --target 10 --cap 9999 --per-task-inflight 3 \ --ak variant=high 2>&1) echo "$out" | sed "s/^/[pass $pass] /" >>$LOG From 9e609891cc02879d558a28f3ba2084b9e3c1f601 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 06:33:56 +0000 Subject: [PATCH 4/5] Give every submit a unique nonce so plateaued cells keep filling Sweep payloads are deduplicated for 24h by a hash of the whole request. A repeated identical submit collapses into the first one: it returns the original response, reports the same trial count, and creates nothing. That is harmless while a cell is filling, because `existing + want` changes every pass and the payload changes with it. It becomes a silent stall the moment a cell plateaus -- ruby-rust-port and stripe-clone both settled on --n-trials 10 and then re-sent a byte-identical request every pass for over two hours, each time reporting ten trials while the task stayed at seven and eight respectively. Every submit now carries a unique ODDISH_EVAL_NONCE, matching what dispatch.sh already did for the terra sweep. --- eval-ops/even_fill.py | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/eval-ops/even_fill.py b/eval-ops/even_fill.py index ee4df14bd..0604c20f4 100644 --- a/eval-ops/even_fill.py +++ b/eval-ops/even_fill.py @@ -25,11 +25,18 @@ trials while still printing "Task submitted!", so the cell never fills and the loop reports success forever. +A sweep payload is deduplicated for 24h by a hash of the whole request, so two +identical submits collapse into one and the second silently creates nothing +while still reporting the original trial count. That bites exactly when a cell +plateaus: once `existing + want` stops changing between passes the payload is +byte-identical and the cell stops filling forever. Every submit therefore +carries a unique ODDISH_EVAL_NONCE. + --ak flags are load-bearing: opencode's `variant` is a CliFlag with NO default, so omitting `--ak variant=high` silently runs a different configuration than the original arm. Pass every knob explicitly, every time. """ -import argparse, json, re, subprocess, sys, datetime +import argparse, json, re, subprocess, sys, time, datetime ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" PENDING = ("pending", "queued", "running", "blocked", "preparing", "submitted", @@ -137,7 +144,8 @@ def main(): n = cells[name]["existing"] + n_new cmd = [ODDISH, "run", "-p", a.dataset, "-t", name, "-a", a.agent, "-m", a.model, "--n-trials", str(n), "-e", "modal", - "-E", a.exp, "--force", "--background"] + "-E", a.exp, "--force", "--background", + "--ae", f"ODDISH_EVAL_NONCE={name}-{time.time_ns()}"] for kv in a.ak: cmd += ["--ak", kv] if a.dry_run: @@ -201,7 +209,8 @@ def main(): n = cells[name]["existing"] + 1 # target-based: creates exactly one cmd = [ODDISH, "run", "-p", a.dataset, "-t", name, "-a", a.agent, "-m", a.model, "--n-trials", str(n), "-e", "modal", - "-E", a.exp, "--force", "--background"] + "-E", a.exp, "--force", "--background", + "--ae", f"ODDISH_EVAL_NONCE={name}-{time.time_ns()}"] for kv in a.ak: cmd += ["--ak", kv] if a.dry_run: From ad8318013c8176f6b6dddc7e515072dbc9639bd2 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 22:47:42 +0000 Subject: [PATCH 5/5] Make opencode compact at 80% of the context window 18 trials died with ContextWindowExceededError. The cause is opencode's compaction threshold, not the model or the task. Measured over the 1,715 steps of rust-java-lsp-1108: max effective context 1,037,624 (99% of gemini-3.7-flash's hard limit) p90 919,663 steps above 900k 212 compaction events 2 opencode runs the conversation to ~99% of the window before compacting, so it sits with no headroom for hundreds of steps and the next sizeable tool result tips the request over. Gemini then rejects it outright: "The input token count exceeds the maximum number of tokens allowed 1048576." code 400, INVALID_ARGUMENT opencode's config has no percentage threshold; compaction.reserved is the token headroom it keeps free. Reserving 209,716 of the 1,048,576-token window leaves 20% free, i.e. compaction at ~80%. Delivering it needs a sweep config rather than --ak: oddish parses agent kwargs into dict[str, str], so a nested value would reach Harbor as a string and break its dict merge. even_fill.py grows --sweep-template, rendering n_trials per submit; the template also carries variant=high, which was previously passed through --ak and must not be lost in the move. Verified on a live trial: /agent_config/kwargs/variant = "high" /agent_config/kwargs/opencode_config = {"compaction": {"auto": true, "reserved": 209716}} Also adds prune_infra.py and audit_exceptions.py. The prune cannot run with the current key -- trial deletion returns 403 "Insufficient scope. Required: full, got: tasks" -- but the script is correct and ready for a key that has it. --- eval-ops/audit_exceptions.py | 82 +++++++++++++++++++++++ eval-ops/even_fill.py | 93 +++++++++++++++++++++----- eval-ops/even_loop.sh | 24 ++++--- eval-ops/oc_sweep.yaml | 26 ++++++++ eval-ops/prune_infra.py | 122 +++++++++++++++++++++++++++++++++++ 5 files changed, 322 insertions(+), 25 deletions(-) create mode 100644 eval-ops/audit_exceptions.py create mode 100644 eval-ops/oc_sweep.yaml create mode 100644 eval-ops/prune_infra.py diff --git a/eval-ops/audit_exceptions.py b/eval-ops/audit_exceptions.py new file mode 100644 index 000000000..317aea7fa --- /dev/null +++ b/eval-ops/audit_exceptions.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +"""Tally why trials actually failed, using Harbor's own classification. + +The authoritative answer is result.harbor_exception.exception_type -- Harbor +names the failure itself. The error_message string is a red herring: it always +opens with "Command failed (exit 1): opencode ..." regardless of cause, and its +middle is elided server-side, so grepping it tells you very little. + +Also counts trials that carry a real verifier result despite the exception, +since those solved the task and are being discarded by the validity rule. +""" +import json, subprocess, sys, collections + +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +EXP = "826d7d88" +PENDING = ("pending", "queued", "running", "blocked", "preparing", "submitted", + "claimed", "in_progress", "initializing", "retrying") + + +def api(args, tries=5): + for _ in range(tries): + try: + r = subprocess.run([ODDISH] + args, capture_output=True, text=True, + timeout=600) + if r.returncode == 0 and r.stdout.strip(): + return json.loads(r.stdout) + except Exception: + pass + return None + + +d = api(["status", EXP, "--json"]) +if d is None: + sys.exit("experiment fetch FAILED") +tasks = {t["id"]: t["name"] for t in d.get("tasks") or []} + +kinds = collections.Counter() +by_task = collections.defaultdict(collections.Counter) +scored_anyway = [] # exception raised, but the verifier still produced a reward +missing = [] + +for tid, tname in tasks.items(): + td = api(["status", tid, "--json"]) + if td is None: + missing.append(tname) + continue + for tr in td.get("trials") or []: + if tr.get("experiment_id") != EXP or tr.get("superseded_by_trial_id"): + continue + if (tr.get("status") or "").lower() in PENDING: + continue + res = tr.get("result") or {} + exc = (res.get("harbor_exception") or {}).get("exception_type") + if not exc: + kinds["(no exception -- clean run)"] += 1 + continue + kinds[exc] += 1 + by_task[tname][exc] += 1 + rew = tr.get("reward") + main = (res.get("main") or {}) + if rew is not None or main: + scored_anyway.append((tname, tr.get("id"), exc, rew, + main.get("passed"), main.get("total"), + (res.get("holdout") or {}).get("pass_rate"))) + +if missing: + print(f"WARNING: {len(missing)} task(s) unreadable: {missing}") + +print("=== why trials ended, by Harbor's own exception_type ===") +for k, v in kinds.most_common(): + print(f" {v:>4} {k}") + +print("\n=== exceptions by task ===") +for tname in sorted(by_task): + row = ", ".join(f"{k}={v}" for k, v in by_task[tname].most_common()) + print(f" {tname:<32} {row}") + +print(f"\n=== trials that raised an exception but STILL have a verifier result " + f"({len(scored_anyway)}) ===") +for tname, tid, exc, rew, passed, total, hold in sorted(scored_anyway, + key=lambda r: -(r[3] or 0)): + print(f" {tid:<40} rew={rew} main={passed}/{total} holdout_rate={hold} {exc}") diff --git a/eval-ops/even_fill.py b/eval-ops/even_fill.py index 0604c20f4..bbcae29e3 100644 --- a/eval-ops/even_fill.py +++ b/eval-ops/even_fill.py @@ -32,9 +32,13 @@ byte-identical and the cell stops filling forever. Every submit therefore carries a unique ODDISH_EVAL_NONCE. ---ak flags are load-bearing: opencode's `variant` is a CliFlag with NO default, -so omitting `--ak variant=high` silently runs a different configuration than the -original arm. Pass every knob explicitly, every time. +Agent kwargs are load-bearing: opencode's `variant` is a CliFlag with NO +default, so omitting variant=high silently runs a different configuration than +the original arm. Pass every knob explicitly, every time. + +Nested kwargs cannot go through --ak at all -- oddish parses those into +dict[str, str], so a dict would arrive at Harbor as a string. Use +--sweep-template for those; see submit_cmd. """ import argparse, json, re, subprocess, sys, time, datetime @@ -56,6 +60,29 @@ def api(args, tries=6): return None +def submit_cmd(a, name, n, nonce): + """Build one submit. Two shapes, because a sweep template is the only way to + pass a nested agent kwarg -- see --sweep-template.""" + base = [ODDISH, "run", "-p", a.dataset, "-t", name, "-e", "modal", + "-E", a.exp, "--force", "--background", "--ae", nonce] + if not a.sweep_template: + cmd = base + ["-a", a.agent, "-m", a.model, "--n-trials", str(n)] + for kv in a.ak: + cmd += ["--ak", kv] + return cmd, None + # Render the template with this submit's n_trials. Written per task so + # concurrent passes cannot race on one shared file. + tpl = open(a.sweep_template).read() + rendered, hits = re.subn(r"(?m)^(\s*n_trials:\s*)\d+\s*$", rf"\g<1>{n}", tpl) + if hits != 1: + raise SystemExit(f"sweep template must contain exactly one n_trials line " + f"(found {hits}) -- refusing to submit") + path = f"/home/user/terra-run/.sweep-{name}.yaml" + with open(path, "w") as f: + f.write(rendered) + return base + ["-c", path], path + + def stamp(): return datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds") @@ -72,8 +99,18 @@ def main(): p.add_argument("--per-task-inflight", type=int, default=0, help="keep this many trials in flight for EVERY task; " "overrides --batch/--cap round-robin mode") + p.add_argument("--min-total-inflight", type=int, default=0, + help="floor on TOTAL in-flight; the neediest tasks are given " + "more than --per-task-inflight to reach it") p.add_argument("--ak", action="append", default=[], help="agent kwarg, repeatable") p.add_argument("--dataset", default="/home/user/terra-run/ds-none") + p.add_argument("--sweep-template", + help="YAML sweep config to submit with (-c) instead of " + "-a/-m/--ak. Required for any agent kwarg that is not a " + "flat string: oddish parses --ak into dict[str, str], so " + "a nested value like opencode_config would reach Harbor " + "as a STRING and blow up its dict merge. n_trials in the " + "template is rewritten per submit.") p.add_argument("--dry-run", action="store_true") a = p.parse_args() @@ -129,6 +166,31 @@ def main(): n_new = min(a.per_task_inflight - c["pend"], a.target - c["held"]) if n_new > 0: want[name] = n_new + + # Even dispersion sets the shape; the floor sets the volume. As tasks + # finish, N-per-task stops being enough to keep the fleet's concurrency + # limit saturated -- with 6 tasks left, 3 each is 18 in flight against a + # limit of 20 plus a queue. Hand the surplus to the tasks with the + # fewest valid trials, since those are the ones that decide when this + # finishes. + if a.min_total_inflight: + room = {n: a.target - c["held"] - want.get(n, 0) + for n, c in cells.items() if c["held"] < a.target} + neediest = sorted(room, key=lambda n: (cells[n]["valid"], cells[n]["held"])) + projected = in_flight + sum(want.values()) + while projected < a.min_total_inflight: + progressed = False + for name in neediest: + if projected >= a.min_total_inflight: + break + if room[name] <= 0: + continue + want[name] = want.get(name, 0) + 1 + room[name] -= 1 + projected += 1 + progressed = True + if not progressed: # every task is capped at its target + break print(f"{stamp()} in_flight={in_flight} per_task_target={a.per_task_inflight} " f"topping_up={len(want)} tasks (+{sum(want.values())}) " f"valid={sum(c['valid'] for c in cells.values())}" @@ -140,14 +202,17 @@ def main(): print("every task already at its in-flight target") return 1 created = 0 - for name, n_new in sorted(want.items(), key=lambda kv: -kv[1]): + # Neediest task first: fewest VALID trials, then fewest held. The + # submits go out serially and the fleet has a fixed concurrency limit, + # so whoever is asked for first gets scheduled first -- that ordering is + # the only lever here for closing the gap between the tasks that are + # nearly done and the ones still near zero. + order = sorted(want.items(), + key=lambda kv: (cells[kv[0]]["valid"], cells[kv[0]]["held"])) + for name, n_new in order: n = cells[name]["existing"] + n_new - cmd = [ODDISH, "run", "-p", a.dataset, "-t", name, "-a", a.agent, - "-m", a.model, "--n-trials", str(n), "-e", "modal", - "-E", a.exp, "--force", "--background", - "--ae", f"ODDISH_EVAL_NONCE={name}-{time.time_ns()}"] - for kv in a.ak: - cmd += ["--ak", kv] + cmd, _ = submit_cmd(a, name, n, + f"ODDISH_EVAL_NONCE={name}-{time.time_ns()}") if a.dry_run: print(f" DRY {name:<32} pend={cells[name]['pend']} " f"existing={cells[name]['existing']} -> --n-trials {n} (+{n_new})") @@ -207,12 +272,8 @@ def main(): # trials in any cell with a past failure, and that cell silently never # fills while the loop reports "ok" forever. n = cells[name]["existing"] + 1 # target-based: creates exactly one - cmd = [ODDISH, "run", "-p", a.dataset, "-t", name, "-a", a.agent, - "-m", a.model, "--n-trials", str(n), "-e", "modal", - "-E", a.exp, "--force", "--background", - "--ae", f"ODDISH_EVAL_NONCE={name}-{time.time_ns()}"] - for kv in a.ak: - cmd += ["--ak", kv] + cmd, _ = submit_cmd(a, name, n, + f"ODDISH_EVAL_NONCE={name}-{time.time_ns()}") if a.dry_run: print(f" DRY {name:<32} held={held} existing={cells[name]['existing']}" f" -> --n-trials {n}") diff --git a/eval-ops/even_loop.sh b/eval-ops/even_loop.sh index c8fd01ead..3a313dee5 100755 --- a/eval-ops/even_loop.sh +++ b/eval-ops/even_loop.sh @@ -2,20 +2,26 @@ # Keep the opencode/gemini arm filling, evenly, until every task holds 10 valid # trials. # -# --per-task-inflight 3 keep three trials running for EVERY task, so all 20 -# advance together instead of one draining the queue. -# Oddish has its own concurrency control, so this does -# not throttle the fleet further -- it just makes sure -# no task sits idle waiting its turn. +# --per-task-inflight 3 keep three trials in flight for EVERY task, so all +# of them advance together instead of one draining +# the queue. Submits go out neediest-first (fewest +# valid trials), which is what closes the gap between +# the nearly-done tasks and the ones near zero. +# --min-total-inflight 40 Oddish runs 20 concurrently, so this keeps roughly +# 20 more queued behind them and the fleet never goes +# idle. As tasks finish, 3-per-task alone stops being +# enough to fill that, and the surplus goes to the +# tasks with the fewest valid trials. # # Note on rate limits: Gemini allows 20M input tokens/minute shared fleet-wide. # An earlier run at 34-35 concurrent had every completion come back # 429 RESOURCE_EXHAUSTED. Watch the valid count -- if it stops climbing while # trials keep finishing, the fleet is back over the ceiling. # -# --ak variant=high is LOAD-BEARING. opencode's `variant` is a CliFlag with no -# default, so dropping it silently runs a different configuration than the arm -# we are trying to reproduce. +# Submits go through oc_sweep.yaml rather than --ak. That file carries BOTH +# variant=high (a CliFlag with no default -- dropping it silently changes the +# configuration) and opencode_config.compaction.reserved, which is nested and +# therefore cannot travel through --ak at all. # # Self-heals from git if the container is recycled mid-run. set -u @@ -41,7 +47,7 @@ for pass in $(seq 1 4000); do # pass resumes) but wastes the poll, so give it room to finish. out=$(timeout 3000 python3 -u even_fill.py 826d7d88 opencode \ google/gemini-3.7-flash --target 10 --cap 9999 --per-task-inflight 3 \ - --ak variant=high 2>&1) + --min-total-inflight 40 --sweep-template $HERE/oc_sweep.yaml 2>&1) echo "$out" | sed "s/^/[pass $pass] /" >>$LOG if echo "$out" | grep -q 'COMPLETE'; then diff --git a/eval-ops/oc_sweep.yaml b/eval-ops/oc_sweep.yaml new file mode 100644 index 000000000..74ac1708a --- /dev/null +++ b/eval-ops/oc_sweep.yaml @@ -0,0 +1,26 @@ +# Sweep config for the opencode/gemini arm. +# +# This exists because --ak cannot carry a nested value: oddish parses agent +# kwargs into dict[str, str], so `--ak opencode_config={...}` would reach Harbor +# as a STRING and its _deep_merge would fail on it. A config file preserves the +# real dict. +# +# compaction.reserved is the token headroom opencode keeps free. Trials were +# overflowing because opencode ran the conversation to ~99% of the window and +# only compacted twice in 1715 steps; a single large tool result then pushed the +# request past gemini-3.7-flash's hard 1,048,576-token limit and Gemini 400'd: +# "The input token count exceeds the maximum number of tokens allowed 1048576." +# Reserving 209,716 keeps 20% free, i.e. compaction at ~80% of the window. +# +# variant: high is load-bearing -- opencode's `variant` is a CliFlag with NO +# default, so dropping it silently runs a different configuration. +agents: + - name: opencode + model_name: google/gemini-3.7-flash + n_trials: 16 + kwargs: + variant: high + opencode_config: + compaction: + auto: true + reserved: 209716 diff --git a/eval-ops/prune_infra.py b/eval-ops/prune_infra.py new file mode 100644 index 000000000..7b323cda4 --- /dev/null +++ b/eval-ops/prune_infra.py @@ -0,0 +1,122 @@ +#!/usr/bin/env python3 +"""Delete infra-failed trials from an experiment (runbook 4b: delete and re-run). + +An infra failure is a harness failure, not an attempt by the model. The runbook +scores only trials that reached a real terminal state, so these are dead weight +-- and worse than dead weight, because the server counts them when resolving +--n-trials N, so every burnt trial permanently occupies a slot against the +target until it is removed. + +Safety rules, in order of how badly each one would hurt to get wrong: + + * Never touch anything still in flight. A pending trial carries an + error_message while it is being retried; deleting it destroys work that was + about to succeed on its own. + * Never touch a VALID trial. Valid means a clean terminal state OR an honest + agent/verifier timeout -- the runbook counts a timeout as a real attempt. + * Refuse the whole pass if any task could not be read. A failed fetch is + indistinguishable from an empty task, and acting on that deletes nothing + but reports success, hiding the failure. + +Classification is by error_message, never by job status: infra failures +routinely report status=success with reward=0 (runbook 3.5). +""" +import argparse, json, subprocess, sys, datetime + +ODDISH = "/home/user/oddish/oddish/.venv/bin/oddish" +PENDING = ("pending", "queued", "running", "blocked", "preparing", "submitted", + "claimed", "in_progress", "initializing", "retrying") +OK_ERR = ("agenttimeouterror", "verifiertimeouterror", "timed out after") + + +def api(args, tries=6): + for _ in range(tries): + try: + r = subprocess.run([ODDISH] + args, capture_output=True, text=True, + timeout=600) + if r.returncode == 0 and r.stdout.strip(): + return json.loads(r.stdout) + except Exception: + pass + return None + + +def stamp(): + return datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds") + + +def main(): + p = argparse.ArgumentParser() + p.add_argument("exp") + p.add_argument("--batch", type=int, default=25, + help="trials per delete call") + p.add_argument("--max", type=int, default=0, + help="stop after deleting this many (0 = no limit)") + p.add_argument("--dry-run", action="store_true") + a = p.parse_args() + + d = api(["status", a.exp, "--json"]) + if d is None: + sys.exit(f"{stamp()} experiment fetch FAILED -- refusing to delete blind") + tasks = {t["id"]: t["name"] for t in d.get("tasks") or []} + + doomed, missing, kept = [], [], {"valid": 0, "pending": 0} + for tid, tname in tasks.items(): + td = api(["status", tid, "--json"]) + if td is None: + missing.append(tname) + continue + for tr in td.get("trials") or []: + if tr.get("experiment_id") != a.exp or tr.get("superseded_by_trial_id"): + continue + st = (tr.get("status") or "").lower() + err = (tr.get("error_message") or "").lower() + if st in PENDING: + kept["pending"] += 1 + continue + if not err or any(k in err for k in OK_ERR): + kept["valid"] += 1 + continue + doomed.append((tname, tr.get("id"))) + + if missing: + sys.exit(f"{stamp()} {len(missing)} task(s) unreadable " + f"({', '.join(missing[:4])}) -- refusing to prune") + + if a.max: + doomed = doomed[:a.max] + per_task = {} + for tname, _ in doomed: + per_task[tname] = per_task.get(tname, 0) + 1 + print(f"{stamp()} infra={len(doomed)} to delete | keeping " + f"valid={kept['valid']} pending={kept['pending']}") + for tname, n in sorted(per_task.items(), key=lambda kv: -kv[1]): + print(f" {tname:<34} {n}") + if not doomed: + return 0 + + ids = [tid for _, tid in doomed] + if a.dry_run: + print(f"{stamp()} DRY RUN -- nothing deleted") + return 0 + + deleted = 0 + for i in range(0, len(ids), a.batch): + chunk = ids[i:i + a.batch] + cmd = [ODDISH, "delete", "--json"] + for tid in chunk: + cmd += ["--trial", tid] + try: + r = subprocess.run(cmd, capture_output=True, text=True, timeout=900) + ok = r.returncode == 0 + except Exception: + ok = False + deleted += len(chunk) if ok else 0 + print(f" batch {i // a.batch + 1}: {len(chunk)} trials " + f"{'deleted' if ok else 'FAILED'}") + print(f"{stamp()} deleted={deleted}/{len(ids)}") + return 0 + + +if __name__ == "__main__": + sys.exit(main())