Tracking: SWE-bench Verified — leaderboard-conformant run (ADR-173)

[ruvnet]

Goal

A conformant SWE-bench Verified (500) placement — top-10 ≈ 70% — at the cost-per-resolve frontier, and the Pareto-frontier crown (target ~73% @ ~$0.15/resolve vs leaders burning $3–5/inst). Ref: ADR-173.

Status: STAGED (gated on Lite proving the pipeline)

• Manifest fetched: 500 instances (verified-500.json) ✅
• Images NOT cached — 0/500; ~750 GB needed vs ~543 GB free → cannot cache concurrently. Verified runs as a streaming pull→solve→prune track, launched once conformant Lite confirms real submits.
• Model: minimax/minimax-m2.5 (75.8% Verified @ ~$0.07/inst on the board) → DeepSeek V3.2 / Kimi K2.5 as alternates.
• Loop: stateful-PTY (ADR-170) for the cleaner Verified env.

Board context (fetched 2026-06-22)

Top ~76.8% (Claude 4.5 Opus); top-10 cutoff ~70%; all via mini-SWE-agent v2.

Method gates

Conformant (no gold oracle in-loop, leakage-guarded); batch-eval + Wilson CI only; --max-cost bounded; $/resolve reported. Frequent progress replies below.

Status: staged behind Lite. Streaming launcher + first pulls next.

[ruvnet]

Status update — Verified deferred pending ADR-177 decision (08:47Z)

The Verified run never started: the Lite conformant ablation (#45, ADR-177) established the scaffold ceiling first, and it's sobering for Verified.

Lite conformant results (gold-graded): cheap models cap 12-16%, frontier (Opus best-of-3) 33% — and even frontier is scaffold-bound (vs Opus's 76.8% Verified via mini-SWE-agent). Every cheap lever (oracle / qwen / Opus-sniper / plan-then-edit) was null.

Implication for this issue: Verified top-10 is ~70%. Our scaffold caps ~33% even with frontier coding on the easier Lite set — so a conformant Verified top-10 is not reachable with the current MCTS+self-repro scaffold, regardless of budget. The original Pareto-crown projection here is falsified.

Verified is deferred until ADR-177's decision (full-300 Opus / build mini-SWE-agent-v2 scaffold / ship product). The manifest (verified-500.json) is committed and ready; the images would need a streaming pull. Keeping open as a placeholder; no Verified spend without an explicit decision + a scaffold capable of clearing ~70%.

[ruvnet]

✅ Verified-500 cascade — measured result

55.6% (278/500), Wilson 95% CI [51.2, 59.9] — official swebench gold eval (run_id verified-500-cascade-local, Total 500 / completed 447 / resolved 278). Conformant (solver never sees PASS_TO_PASS/FAIL_TO_PASS).

• Method: GLM→Opus empty-patch cascade (same as the Lite #453 submission) — cheap GLM-5.2 base on all 500; only empty-patch give-ups escalate to Claude-Opus-4.8 (167 escalations).
• Cost ~$0.15/instance (estimate; per-instance cost not captured — to be tightened before any leaderboard submission).
• Beats the Lite cascade (51.3%) — consistent with Verified being human-validated/cleaner.
• Artifacts: submissions/swe-bench-verified/darwin-glm-opus-cascade/ (preds + official results.json). LEARNINGS §47.

Lifts Verified from the old single-traj 46.4% to a competitive 55.6% at the cost-Pareto frontier.