… branch, preflight clean-tree (#38) Address final whole-branch review Important findings (minimal scope) + 2 Minors: - Fix 1 (Important #3): score_predictions raises RuntimeError on scorer returncode!=0 BEFORE parse_run_summary, so a scorer crash (missing CJK toolchain, numpy-pin violation, OOM) surfaces instead of silently emitting zero/None metrics and a confusing gate verdict. - Fix 2 (Important #2): _wait_ci probes gh pr view --json state once first; a MERGED state or non-zero (branch gone) returns immediately instead of polling the full 900s on a crash-and-resume where gh pr merge --delete-branch already ran. Live-branch pending/pass/fail/timeout logic preserved. - Fix 3 (Important #1, minimal): release() preflight clean-tree check at the top (BEFORE eval) so a dirty tree doesn't waste a 4h eval with a SHA that won't match the evaluated code; docs/RELEASE.md crash-recovery section listing publish_release's step order + manual completion recipe. - Fix 4 (Minor #5): comment that tok_per_sec=None is intentionally deferred to the speed workstream (gate speed Check is advisory/inert). - Fix 5 (Minor #8): flag baseline_was_override=true on the manifest when the selected baseline had gate.verdict==OVERRIDE (informational; no gate change). Tests: +8 (scorer rc!=0 raises; rc=0 passes; merged short-circuit; live-branch poll; dirty-tree preflight rejects before eval; clean-tree passes; override baseline flagged; clean baseline unflagged). Existing _wait_ci tests updated for the new merged-detection probe. Full idempotent resume, backend-aware looping detector, PAT rotation deferred to workstream #1. Co-authored-by: Claude <noreply@anthropic.com>

Benchmark on your GPU — help us build a leaderboard #8

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions