Add the eval-ops toolkit used to drive the SWE-Marathon sweeps - #1269
Merged
Conversation
Operational scripts for running the SWE-Marathon evals end to end, collected here so they survive a container recycle and so the next run does not have to rediscover the same gotchas. Nothing here is imported by the CLI, the backend, or the frontend; it is a standalone toolkit under eval-ops/. What it covers: - Filling an experiment to N trials per task. Sweeps are target-based, so --n-trials N creates N - existing and re-running a pass is never a double submit. even_fill.py tops up fewest-held-first under a global in-flight cap and a per-pass batch size, so all 20 tasks advance together rather than the scheduler draining one; even_loop.sh repeats it until every cell is at target. - Throttling to what the providers actually allow. Gemini's input-token quota is shared fleet-wide: at 34-35 concurrent trials every completion came back 429 RESOURCE_EXHAUSTED, and each 429 makes opencode retry with the full context re-sent, inflating usage further. At ~15 concurrent, completions come back clean. The CUA-verifier tasks have their own cap, handled by cua_fill.py. - Classifying outcomes by error_message rather than job status, because infra failures routinely report status=success with reward=0. poll_all.py and oc_poll.py both use that rule; "retrying" is a pending state, not a failure, so a cleanup pass does not destroy trials that were about to succeed. - passk.py computes pass@k with the same estimator as the frontend's pass-at-k.ts, scoring only trials that reached a real terminal state, and refuses to score an experiment whose tasks it could not all read. - clear_broken.py deletes only zero-token infra failures, keeps trials that consumed tokens, and skips anything still in flight. Agent flags are passed explicitly everywhere: opencode's `variant` is a CliFlag with no default, so omitting it silently runs a different configuration than the arm being reproduced.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The filler asked the server for `held + 1` trials, where held = valid + pending. But the server resolves --n-trials N by subtracting every trial it still holds for the cell, and it counts by status -- and infra failures report status=success with reward=0 and a populated error_message. Every terminal trial in the opencode arm has status=success, the 30 burnt ones included. So in any cell with a past failure, `held + 1` was less than or equal to what the server already had, and the submit created nothing while still printing "Task submitted!". excel-clone sat at held=0 with one burnt trial and got re-submitted on three consecutive passes, creating zero trials each time; the cells with eight burnt trials would never have filled at all. Track `existing` (everything the server still has) separately from `held` (progress toward the target), submit existing+1, and parse the trial count back out of the response so a divergence between the two notions is visible in the log rather than silent.
Bugbot is paused — on-demand spend limit reachedBugbot uses usage-based billing for this team and has hit its on-demand spend limit. A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue. |
Round-robin with a global cap keeps every task advancing, but it leaves tasks idle whenever the cap binds -- four consecutive passes submitted nothing at all because in-flight was already at the ceiling, so nothing progressed. --per-task-inflight N instead keeps N trials running for every task and lets Oddish's own concurrency control decide what actually executes. The per-task count is clamped to what the target still needs, so a cell close to its target does not overshoot on the last top-up. The created-count check earns its keep here: topping ruby-rust-port up by 3 created 10, meaning the server's notion of "existing" for that cell did not match ours. The pass flags any such divergence inline rather than reporting a uniform success. Also raises the per-pass timeout: a per-task pass polls 20 tasks and then issues up to 20 submits, each uploading the dataset, which does not fit in the window sized for a three-submit round-robin pass.
Sweep payloads are deduplicated for 24h by a hash of the whole request. A repeated identical submit collapses into the first one: it returns the original response, reports the same trial count, and creates nothing. That is harmless while a cell is filling, because `existing + want` changes every pass and the payload changes with it. It becomes a silent stall the moment a cell plateaus -- ruby-rust-port and stripe-clone both settled on --n-trials 10 and then re-sent a byte-identical request every pass for over two hours, each time reporting ten trials while the task stayed at seven and eight respectively. Every submit now carries a unique ODDISH_EVAL_NONCE, matching what dispatch.sh already did for the terra sweep.
18 trials died with ContextWindowExceededError. The cause is opencode's
compaction threshold, not the model or the task. Measured over the 1,715 steps
of rust-java-lsp-1108:
max effective context 1,037,624 (99% of gemini-3.7-flash's hard limit)
p90 919,663
steps above 900k 212
compaction events 2
opencode runs the conversation to ~99% of the window before compacting, so it
sits with no headroom for hundreds of steps and the next sizeable tool result
tips the request over. Gemini then rejects it outright:
"The input token count exceeds the maximum number of tokens allowed 1048576."
code 400, INVALID_ARGUMENT
opencode's config has no percentage threshold; compaction.reserved is the token
headroom it keeps free. Reserving 209,716 of the 1,048,576-token window leaves
20% free, i.e. compaction at ~80%.
Delivering it needs a sweep config rather than --ak: oddish parses agent kwargs
into dict[str, str], so a nested value would reach Harbor as a string and break
its dict merge. even_fill.py grows --sweep-template, rendering n_trials per
submit; the template also carries variant=high, which was previously passed
through --ak and must not be lost in the move. Verified on a live trial:
/agent_config/kwargs/variant = "high"
/agent_config/kwargs/opencode_config = {"compaction": {"auto": true,
"reserved": 209716}}
Also adds prune_infra.py and audit_exceptions.py. The prune cannot run with the
current key -- trial deletion returns 403 "Insufficient scope. Required: full,
got: tasks" -- but the script is correct and ready for a key that has it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
eval-ops/, the operational scripts used to run the SWE-Marathon eval sweeps end to end. Nothing here is imported by the CLI, the backend, or the frontend — it is a standalone toolkit, checked in so it survives a container recycle and so the next run does not have to rediscover the same gotchas.What's in it
Filling an experiment. Sweeps are target-based (
--n-trials NcreatesN - existingfor a given task/agent/model/experiment), so re-running a pass is never a double submit.even_fill.pysupports two shapes: round-robin under a global in-flight cap, and--per-task-inflight Nwith a--min-total-inflightfloor, which keeps N trials running for every task and hands the surplus to the tasks with the fewest valid trials.even_loop.shrepeats a pass until every cell is at target.dispatch.sh,cycle.sh, andrun_forever.share the equivalents for the codex/terra effort sweep.Three counting traps this encodes, each of which silently stalled the fill in practice:
--n-trials, including burnt ones — and infra failures reportstatus=success. Submittingheld + 1therefore resolves to zero new trials in any cell with a past failure, while still printingTask submitted!.ODDISH_EVAL_NONCE.--akparses agent kwargs intodict[str, str], so a nested value would reach Harbor as a string and break its dict merge. Nested config goes through--sweep-templateinstead.Diagnosis by Harbor's own classification.
audit_exceptions.pytalliesresult.harbor_exception.exception_typerather than greppingerror_message— every failure shares the sameCommand failed (exit 1): opencode ...prefix regardless of cause, and its middle is elided server-side, so the string cannot distinguish a rate limit from a context overflow. On the current arm that split is 63ApiRateLimitError/ 18ContextWindowExceededError/ 4AgentTimeoutError, which the error string alone would not have shown.Context management.
oc_sweep.yamlsetscompaction.reservedso opencode compacts at ~80% of the context window. Trials were overflowing because opencode ran the conversation to ~99% of the window and compacted only twice in 1,715 steps; a single large tool result then pushed the request pastgemini-3.7-flash's hard 1,048,576-token limit and Gemini rejected it withINVALID_ARGUMENT.Classifying outcomes by
error_message, not job status. Infra failures routinely reportstatus=success, reward=0with a populatederror_message.poll_all.pyandoc_poll.pyboth use that rule.retryingis treated as a pending state rather than a failure, so a cleanup pass does not destroy trials that were about to succeed on their own.Scoring.
passk.pycomputes pass@k with the same estimator asfrontend/src/lib/pass-at-k.ts, counting only trials that reached a real terminal state (a clean run or an honest agent/verifier timeout), and refuses to score an experiment whose tasks it could not all read — a silently dropped task shifts the result.Cleanup.
clear_broken.pydeletes only zero-token infra failures.prune_infra.pydeletes infra-classified trials wholesale, never touching anything in flight or any trial that reached a real terminal state, and refusing the pass entirely if any task could not be read.Note
Agent flags are passed explicitly everywhere. opencode's
variantis aCliFlagwith no default, so omitting it silently runs a different configuration than the arm being reproduced — that cost a full 200-trial resubmission once already. It now travels in the sweep template alongside the compaction config.Testing
No automated tests: these are operational scripts that talk to the live Oddish API, and the repo's suite does not cover
eval-ops/. They have been exercised against the running SWE-Marathon experiments — the even-fill loop is currently driving the opencode/gemini arm,passk.pyproduced the reported pass@1/3/8 numbers for the six-rung terra effort sweep, and the compaction config was verified on a live trial before being wired in:prune_infra.pyis the one script not exercised end to end: trial deletion returns403 Insufficient scope. Required: full, got: taskswith the current key, so only its dry-run path has run.