run_sweep.sh re-runs whole targets to clear a few transient failures — retry per-problem instead

[aallan]

Problem

scripts/run_sweep.sh's is_clean marks a target dirty on any transient row, and do_target then re-runs the entire target (all 60 problems). For a target that is 57/60 good with 3 transient timeouts, that's 57 redundant runs — expensive on slow models (kimi at parallel 3 is the pathological case).

Worse under a flaky provider: a full re-run re-rolls all 60 problems against the flaky API, so it can introduce new transient timeouts on problems that were previously fine, and may not converge within SWEEP_RETRIES (2) — leaving the target dirty after two full passes.

Surfaced live during the v0.0.16 sweep (the PR #100 tooling): moonshot-kimi-k2.6-...-vera had 3 transient timeouts and the sweep unlinked the 60-row file and re-ran all 60.

Why the surgical tool doesn't already solve it

scripts/rerun_failed.py re-runs only the failed problems and splices them back by problem_id. But it cannot run concurrently with the sweep — both unlink/write the same canonical file, so a concurrent splice races the sweep's whole-target re-run. Today you must choose: the sweep (wasteful whole-target re-run) or surgical (manual, and only safe once the sweep has finished the target).

Proposed fix

Teach run_sweep.sh's dirty path to retry only the failed problems — fold rerun_failed.py's per-problem re-run + splice into do_target — instead of re-running the whole target. Reserve the full re-run for incomplete coverage (<60 unique problem_ids, i.e. a killed/partial target that genuinely needs all problems).

Concretely:

• Target dirty because coverage < expected (partial/killed file) → full run, as today.
• Target dirty only because of transient rows on an otherwise-complete file → re-run just those problem_ids into a scratch dir and splice, looping up to SWEEP_RETRIES.

Benefit

• Far less wasted compute on slow models (3 re-runs, not 60).
• Better convergence under provider flakiness — you re-roll only the failures, not the whole set (a full re-run can newly break previously-good problems).
• Removes the sweep-vs-surgical either/or: the sweep becomes surgical by default.

References

• scripts/run_sweep.sh — is_clean, do_target
• scripts/rerun_failed.py — failed_pids, rerun_one, splice (the logic to reuse)

🤖 Generated with Claude Code