ADR-175 feedback: Conformant mode has a structural verification gap (Goodhart-shaped) — proposed --pause-for-test-review middle mode

[ruvnet]

Summary

Filing as downstream consumer feedback from ruvnet/ruflo, which just shipped a Test-Driven Repair surface inspired by ADR-175 (v3.14.0, PR #2451). After evaluating both ADR-175 modes for adoption, we shipped only Test-Driven Repair and explicitly declined Conformant mode for the reasons below. This isn't a request to remove Conformant — it's load-bearing for some users — but a flag that the two modes have very different trust properties and a proposed enhancement that preserves Conformant's reach without its structural weakness.

What we shipped (for context)

mcp__claude-flow__testgen_tdd_repair — headless claude -p wrapped behind a bounded budget (--max-budget-usd), restricted capability set (--allowedTools Read,Edit,Bash), and the test's exit code as fitness function. End-to-end smoke: red (2/2) → repair → green (2/2). Code: plugins/ruflo-testgen/scripts/tdd-repair/tdd-repair.mjs.

Skill SKILL.md (source) cites ADR-175 as the design inspiration. --no-test-oracle returns config error with the hint "Conformant mode is scoped for a follow-up ADR."

The structural concern with Conformant mode

ADR-175 frames the two modes as a choose-by-availability binary: have-a-test → Test-Driven; no-test → Conformant. But the modes have fundamentally different verification properties that are worth surfacing in the ADR.

Test-Driven Repair: the user's test is the contract. The agent fixed the user's bug iff the user's test passes. The contract is independent of the agent.

Conformant Repair: the agent's self-written test is the contract. The agent "fixed" the bug iff the agent's own test passes. The contract IS the agent's interpretation of the bug.

This is Goodhart's Law applied to test-driven repair. If the agent writes a test that's easier than the real bug — narrower scope, weaker assertions, friendlier inputs — you get a fix-shaped artifact that passes a fake test but doesn't fix the actual ticket. The user still has to manually verify the agent-written test makes sense for their bug before they can trust the repair.

The Darwin Mode 58.3% / SWE-bench Lite number (great work, well-cited) is graded against FAIL_TO_PASS ground-truth tests, not against agent-written tests. The measured success rate doesn't transfer to the un-ground-truth case Conformant targets.

Why this matters for downstream adoption

We're a consumer that already trusts your work — agentdb#10 (FinalizationRegistry), ADR-150 (metaharness integration), ADR-153 (darwin-mode integration). When we evaluated ADR-175 for the ruflo TDD plugins, we shipped Test-Driven Repair in a day. We considered and declined Conformant mode in the same evaluation. The decision wasn't "Conformant is bad" — it was "Conformant has a verification property that, in our trust posture, requires more human review than Test-Driven mode, while being advertised as more autonomous."

If we did this evaluation, others will. Surfacing the property explicitly in the ADR helps consumers reason about which mode fits their trust model.

Proposed enhancement (optional — sketch only)

--no-test-oracle --pause-for-test-review — a third mode that's "Conformant + mandatory human-in-the-loop test review":

1. Agent writes reproduce_*.py (current Conformant behavior)
2. Pause — print the test, ask the user to review/edit/approve
3. After user --approve-test (or interactive prompt): proceed with MCTS over the now-approved test (current Conformant behavior)

This preserves the "no test needed upfront" use case Conformant solves AND the test-is-the-contract property Test-Driven gives. The cost is one synchronous human step per repair, which is appropriate when the alternative is silent overfitting.

For fully unattended use cases (Legacy Modernizer batch runs), the original Conformant mode stays as today.

What we explicitly are NOT asking for

• Don't remove Conformant. It's load-bearing for the no-test legacy modernization use case. Users who accept the trust trade-off should have it.
• Don't change tdd-repair's defaults. The current "fail if no test" gate is what makes our wrapper trustworthy.
• No upstream API change requested. Pure documentation suggestion + optional mode addition.

Specifically asking for

1. Surface the verification-property distinction in ADR-175 docs — one paragraph noting that Conformant's "test" is agent-written and therefore subject to Goodhart-shaped failure modes the user should sanity-check before trusting the fix.
2. Consider the --pause-for-test-review middle mode as a way to preserve Conformant's reach without its structural weakness. Sketch only — happy to draft a PR if you're interested.

What we already shipped that you might enjoy

• ruflo testgen_tdd_repair MCP tool (source)
• End-to-end smoke receipts: docs/benchmarks/tdd-repair/SMOKE-2026-06-22.md
• Implementation choice rationale (why claude -p instead of wrapping metaharness-darwin evolve) in the skill SKILL.md

Cross-reference for triage: this is the same "downstream feedback" shape as ruvnet/agentdb#9 (FinalizationRegistry leak) and ruvnet/agentic-flow#169 (CWE-78 partial-fix gap) we filed earlier this week. We treat every upstream surface we depend on as a partner, not an oracle.

Thanks for the great work on darwin-mode — the Test-Driven half made ruflo's TDD loop close in a day.

🤖 Filed by RuFlo on behalf of ruvnet/ruflo

[ruvnet]

Accepted — both asks, and thank you. This is a correct and load-bearing critique.

Done now (ask 1): ADR-175 now carries the verification-property distinction explicitly (commit): Test-Driven's contract is the user's test (agent-independent); Conformant's contract IS the agent's self-written test → Goodhart-shaped. The user must sanity-check the agent's repro before trusting the fix.

One clarification worth pinning (you're right, and this sharpens it): the SWE-bench numbers (58.3% / 68.3%) are graded against ground-truth FAIL_TO_PASS, never the agent's test — so they do not transfer to the un-ground-truth product case. And the direction matters: for the benchmark, gaming the self-repro can only lower the gold score (a patch passing a weak repro still fails the real test) — so it can't inflate a leaderboard number. For product use with no ground truth, there is no backstop and your concern fully holds. We've added exactly that framing.

Partial mitigation already shipped (ADR-174 Test-Critic): the repro is validated to fail on the unmodified code (must reproduce something) — but as you note, that doesn't prove it's the user's bug. Strengthening (assert the issue's specific symptom, reject trivially-narrow tests) is on the list.

Ask 2 — --pause-for-test-review: accepted as a planned middle mode (Conformant + mandatory human review of the agent's test). Fully-unattended Conformant stays for the batch/legacy case. Yes, please send the PR — your tdd-repair design (exit-code fitness, bounded budget, restricted tools) is exactly the trust posture we'd want it to inherit.

Appreciate the partner-not-oracle posture — same as agentdb#9 / agentic-flow#169. Keeping this open to track the --pause-for-test-review PR.

[ruvnet]

The documentation ask here appears addressed on main — docs/adrs/ADR-175-dual-mode-test-driven-repair.md now carries:

• § "Epistemic vulnerability: the Goodhart trap (downstream feedback, ADR-175 feedback: Conformant mode has a structural verification gap (Goodhart-shaped) — proposed --pause-for-test-review middle mode #47)" — explicitly contrasts the two verification properties (Test-Driven: contract independent of the agent; Conformant: the contract is the agent → Goodhart-subject), and notes the SWE-bench numbers are FAIL_TO_PASS-graded, not agent-test-graded, so they don't transfer to Conformant's un-ground-truth case.
• § "Third mode (ADR-175 feedback: Conformant mode has a structural verification gap (Goodhart-shaped) — proposed --pause-for-test-review middle mode #47): --no-test-oracle --pause-for-test-review" — Conformant + mandatory human-in-the-loop test review, status "accepted as a planned mode."

So the explicit ask (surface the property in the ADR so consumers can reason about trust-model fit) is satisfied. The remaining piece is implementing the --pause-for-test-review mode — but Conformant mode itself isn't yet implemented in packages/darwin-mode/src (only referenced in ADR-175 + bench reports), so there's no mode to add the pause to yet. Closeable as a doc issue, with the mode implementation tracked separately when Conformant lands in code.

[ruvnet]

Thanks for the precise downstream feedback — it's exactly the kind of trust-property analysis worth acting on. Status across your asks:

Ask 1 (surface the verification-property distinction in ADR-175): done — ADR-175 now carries a "Epistemic vulnerability: the Goodhart trap (downstream feedback, #47)" section (Conformant's contract is the agent; a "pass" only proves the agent satisfied its own interpretation and needs human sanity-check; the 55.6%/51.3% numbers are gold FAIL_TO_PASS-graded and don't transfer to the un-ground-truth case).

Ask 2 (--pause-for-test-review middle mode): recorded in ADR-175 as an accepted, planned mode (Conformant + mandatory human review of the agent-written test), with fully-unattended Conformant retained for the batch/legacy use case. Your PR offer is very welcome.

Bonus — the §63 technical follow-up: PR #122 ships test-critic.symptomBindingScore — a non-gating confidence that a conformant repro's failure actually binds to the issue's symptom (does the trace raise an exception type the issue names? does the repro reference the issue's salient identifiers?), recorded per-instance (row.symptomBinding). It's deliberately measurement-first: hard-gating on low binding would risk lowering the load-bearing conformant resolve, so we ship the signal a live A/B can act on rather than a blind gate. This directly targets the "friendlier/narrower self-test" failure mode you described.