Tracking: SWE-bench Lite — leaderboard-conformant run (ADR-173, MiniMax M2.5)

[ruvnet]

Goal

A legitimate SWE-bench Lite placement (top-10 ≥45% → #1 >60.33%) via the conformant --no-test-oracle solver at the cost-per-resolve frontier. Ref: ADR-173, docs/LOOP_WORKER.md.

Why this is separate from our 68.3%: that number used the gold FAIL_TO_PASS as an in-loop oracle → not submittable. This run never touches the gold tests during solving (Docker conformant signal, ADR-173 L0.5/L0.6); the gold harness scores once at the end. leaderboardConformant is asserted in the report.

Current run (L1)

• Model: minimax/minimax-m2.5 · agentic loop, max-steps 20, concurrency 3, --max-cost 60
• Signal: existing tests in the changed package, run inside the instance Docker image (deps present, no gold patch)
• Progress: 3/300, agent submitting, ~$0.037/instance, proc alive
• Targets: ≥45% = top-10, >50% = top-5, >60.33% = feat(web-ui): browser-based agent harness generator + Claude skill/agent/command authoring #1

Board context (fetched 2026-06-22)

#1 ExpeRepair+Claude-4-Sonnet 60.33% · top-10 cutoff ~45%. Board skews 2024–2025.

Method gates

Only batch-eval numbers (Wilson 95% CI) reported; --max-cost bounds spend; $/resolve tracked. Frequent progress replies below.

Status: L1 generating. Batch eval + first conformant number on completion.

[ruvnet]

Progress — 20:07Z

• 4/300 generated · 2 submits · proc alive
• run cost $0.12 (~$0.030/inst) of the $60 cap · MiniMax M2.5
• signal: existing-tests-in-Docker (conformant, no gold patch); inloop=false is normal on hard instances (agent submits its best, gold batch decides)
• next: continue generation → official batch eval at 300/300 → resolved/300 + Wilson CI + leakage assert

[ruvnet]

Progress — 20:17Z

• 19/300 · 3 submits (16% submit-rate) · proc alive
• run cost $0.33 / $60 cap (~$0.017/inst) · MiniMax M2.5
• health: disk 543G, no docker hangs
• batch eval pends 300/300 (only the gold batch number is authoritative)

[ruvnet]

Architecture pivot (ADR-174) — 20:22Z

The linear-loop conformant run (MiniMax M2.5) is showing a low submit-rate — a single linear loop hits a context-collapse wall and won't reach the top. Pivoting to the road-to-#1 stack (ADR-174):

1. Base → DeepSeek-V4-Flash ($0.09/$0.18/M, verified on OpenRouter — cheapest reasoning).
2. Test-Critic oracle (L0.6, built+validated): agent writes reproduce_bug.py, validated to FAIL on the unmodified repo inside the instance Docker image (conformant). Plumbing validated $0; first live instance: VALID repro on attempt 1.
3. Conformant MCTS (L2): k=5–10 Docker-forked branches gated by the repro test (the recall lever).
4. Opus-4.8 sniper (L3): repro-gated escalation on the ~10% bedrock; total <$50.

Now measuring the linchpin: repro-validity rate of DeepSeek-V4-Flash over 8 instances. Targets unchanged (top-10 ≥45% → #1). Only batch-eval numbers will be claimed.

[ruvnet]

Linchpin measured (corrected) — 20:33Z

Repro-validity of DeepSeek-V4-Flash: 6/8 valid failing repros (the Test-Critic oracle, ADR-174 L0.6).

• astropy: 6/6 VALID on attempt 1 — the cheap model writes clean bug-reproducing tests reliably.
• django: 0/2 — repro needs test-settings/DB setup (known harder; a django-aware repro template is the fix).
• cost: ~$0.001/instance. (First pass read 25% — that was a classifier bug counting 'ValueError' tracebacks as collection errors; fixed to parse the pytest summary line → real rate ~75%.)

Implication: the conformant oracle is viable → L2 Conformant MCTS is GO (k candidate patches, repro-gated select).

Separately, the MiniMax linear-loop baseline (this issue's original run): 42/300, ~7% submit-rate — confirms a linear loop on a cheap model flails; the search stack is the fix. Only batch numbers claimed.

[ruvnet]

L2 Conformant MCTS — validated end-to-end (20:40Z)

First real-instance run of solve-mcts.mjs (astropy-12907, DeepSeek-V4-Flash, k=3):

• repro=valid (Test-Critic), 1/3 candidate patches passed the repro → fixing patch selected + emitted (500 chars)
• sniper not needed · conformant=true (gold tests never touched) · $0.028 · 272s

The whole architecture works on a live Docker instance: Test-Critic oracle → k candidates → repro-gated select → conformant emit. ~$0.03/inst at k=3 (→ ~$0.05 at k=5).

Driver decision (Path A/B): Nemotron-3-Super-120B :free repro-validity pilot in flight — slow on the free tier (1/8 so far, VALID att 1). DeepSeek-V4-Flash already proved 75% and the MCTS works with it (above), so Path B is a safe fallback if Nemotron's free tier rate-limits. Next: full conformant MCTS batch once the driver locks. Only batch-eval numbers claimed.

[ruvnet]

Linear-loop baseline closed + driver gate update — 20:48Z

Killed the MiniMax M2.5 linear-loop run at 56/300 — submit-rate held at ~5% (3/56). Conclusion locked: a single linear agentic loop on a cheap model flails (won't submit), confirming ADR-174's premise. The MCTS search engine (validated end-to-end above) is the path, not the linear loop — freeing resources for it.

Nemotron driver gate: 3 valid / 5 of 8 so far (astropy mostly VALID att-1; astropy-6938 invalid = '2 errors during collection', correctly caught). Free tier is slow but the validity looks on-track. Final 6/8 verdict pending; DeepSeek-V4-Flash (proven 75%, fast) remains the practical driver for the scaled batch regardless.

Next: lock driver → launch first conformant MCTS pilot (k=5) → batch eval = first submittable number.

[ruvnet]

MCTS pilot interim (9/25) — bottleneck identified (21:02Z)

• repro-valid 7/9 (78%) — DeepSeek-V4-Flash writes valid failing repros reliably (non-django).
• branch-passed only 1/9 — the k=5 candidate patches rarely make the repro PASS. The bottleneck is patch quality, not test generation.
• DeepSeek-V4-Flash here drove BOTH repro + patching → clean baseline: the cheap model can verify but not yet fix.
• Next lever (matches the ADR-174 design): keep DeepSeek for the Test-Critic, swap MCTS patch-gen to MiniMax M2.7 (the intended heavy-lifter, $0.25/$1.00). Possibly more steps/branch (agentic per-branch vs single-shot).
• Pilot finishing → batch eval gives the real conformant resolve-rate (the DeepSeek-only floor). Only batch numbers claimed.

[ruvnet]

Conformant MCTS floor — FIRST authoritative number (21:28Z)

DeepSeek-V4-Flash only, k=5, gold batch eval: 5/25 = 20.0% [Wilson95: 8.9%, 39.1%] · conformant=true · $0.587.

Diagnosis (the loss is fixable):

• 14/25 empty patches — single-shot DeepSeek produced nothing applicable (the dominant loss).
• of the 11 non-empty patches, 5 resolved (45%) — when it writes a real patch, it's often right.
• repro-valid 17/25 (the Test-Critic is fine); branch-passed-in-loop 2 (3 more resolved as best-effort fallback).

So the floor is bounded by patch generation, not reasoning. Next lever (already in code): --patch-model minimax/minimax-m2.7 (the heavy-lifter) + the Opus-4.8 sniper. Launching the MiniMax-patch variant on the same 25 now → measures the recall lift. Below the 45% top-10 bar as expected for the floor; the levers are what close it.

[ruvnet]

SOTA odds (now standing in every loop status) — 21:34Z

First-order: overall-resolve ≈ attempt-rate × conditional-resolve. Floor 0.44×0.45≈20.0% ✓.

target	threshold	odds (DeepSeek floor)
Pareto cost crown	best resolve/$	~55–70% ← the realistic SOTA (10–15× cheaper at any resolve)
Top-10 Lite	≥45%	~35–45%
#1 Lite	>60.33%	~15–20%
Absolute SOTA	~80–85%	<5–10%

The honest wall: 45% conditional-resolve is a ceiling — filling empty patches alone caps ~45%; 70%+ needs MiniMax to write better patches, not just more. Recomputed from the pending MiniMax-M2.7-patch batch (1/25 running). Only measured numbers move these.

[ruvnet]

MiniMax-M2.7 patch swap — FALSIFIED (22:13Z)

Gold batch, same 25-instance pilot, DeepSeek Test-Critic + MiniMax M2.7 patching:

metric	DeepSeek floor	MiniMax M2.7
Resolved/25	5/25 = 20.0%	5/25 = 20.0% (no change)
Non-empty patches	44%	48%
Conditional-resolve	45%	42%
Cost	$0.59	$1.29 (2.2×)

A stronger, 2.2×-pricier patch model moved nothing. Both models leave ~50% of patches empty → the bottleneck is structural: the brittle single-shot search/replace applier, not model capability. Decision: keep DeepSeek for patches.

Next lever (the only one the data supports): a robust patch applicator — agentic read-then-edit per branch instead of single-shot search/replace. Lifts attempt-rate for both models at ~zero extra model cost.

Odds downgraded (honest): Pareto crown ~30–45%, top-10 ~20–30%, #1 ~8–12%, absolute <5%. Two batches both at 20.0% — the floor is real; the applicator fix is what could move it.

[ruvnet]

Definitive DeepSeek-only conformant ceiling — 02:08Z

Three gold-graded 25-instance Lite pilots (conformant):

config	attempt-rate	repro-valid	gold resolve
search floor	44%	68%	5/25 = 20.0% [8.9,39.1]
line-applicator	80%	64%	4/25 = 16.0%
line + repro-gap fix	76%	80%	3/25 = 12.0% [4.2,30.0]

Maxing the plumbing gave ZERO resolve lift — reasoning is the wall. (CIs overlap → the three are statistically indistinguishable at n=25, ~15% center; the honest claim is 'no lift', not '12<20'.)

Goodhart signal (validates #47): combined-resolved is a STRICT SUBSET of floor-resolved (lost django-14411, seaborn-3190). More self-repro gating selected patches passing the agent's weak test but failing gold — a weak model can't author a faithful oracle.

Verdict: pure-cheap can't reach top-10. Launching the Opus-4.8 sniper hybrid (ADR-176): cheap agents gather evidence + fail, Opus fixes the tail. This is the hybrid Pareto test. Odds: pure-cheap top-10 now <10%; hybrid top-10 depends on this run.

[ruvnet]

Ablation Arm D — the frontier ceiling (02:52Z)

Opus critic + Opus coder, conformant: 6/18 = 33.3% [Wilson 16.3, 56.3] (capped at 18 by max-cost).

vs cheap ceiling 12-20%. Frontier coding ~doubles it and solved 2 instances cheap models never got (matplotlib-23314, django-14534) — but still caps ~33%, far below Opus's 76.8% on the Verified leaderboard.

Key finding: the binding constraint is the conformant scaffold (self-repro Goodhart gating + our MCTS/line-applicator), not just model tier. Even Opus's correct patches get displaced when they don't pass the self-written repro. → Top-10 (45%) needs scaffold levers (SBFL localization, plan-then-edit, stronger/multi-repro gating, mini-SWE-agent idioms), not a bigger model.

Matrix completing: A=DS+DS 12% · A′/B/C (qwen/Opus-critic combos) golds pending. (n=18, CI wide — '2x' suggestive; the +2 concrete.)

[ruvnet]

Ablation COMPLETE — the coder binds, not the oracle (04:05Z)

Gold-graded conformant matrix (critic × coder):

critic / coder	DeepSeek	qwen3-coder	Opus
DeepSeek	12%	0%	—
Opus	16% [6.4,34.7]	4%	33% [16.3,56.3]

Cheap-Pareto thesis FALSIFIED. Opus-oracle + cheap-coder (16%) does NOT approach frontier (33%). A strong Opus repro lifts the cheap coder only 12→16% (noise) — DeepSeek patches pass even an Opus-authored repro but still fail gold. The cheap model can't write the cross-file fix, contract or not. qwen3 catastrophic (0-4%, doesn't transfer from EntroPO's harness). Even frontier caps at 33% → the scaffold is also a ceiling.

Next (budget-viable): asymmetric hybrid — Opus oracle + cheap DS coder + Opus sniper on the DS-failure tail (~$30 vs Opus-on-all $63/18). Then scaffold levers (SBFL, plan-then-edit) to lift the 33% ceiling toward 45%. SOTA-at-pennies is dead; the realistic play is competitive resolve cheaper than pure-frontier.