SpecFlow 2.0: local spec-refinement plugin (skills + a small comparison CLI)

[awrobel-gd]

Summary

A local spec-refinement loop shipped as a second marketplace plugin. Independent subagents each read the spec under a different adversarial lens, and each one first draws its own cross-product of the decisions its angle implies and then has to account for every intersection. Where two readings answer the same question differently, or where a lens reaches a cell its axes forced into existence and reports that the spec cannot settle it, that is the finding. Every judgment about whether the spec is sound stays with the model, stated out loud rather than dressed up as a check.

Entrypoint

mcp_server/services/refine_compare.py — the module docstring states the boundary the whole design rests on, and the comment where novelty used to live states the one it refuses to cross. Wired into the CLI by refine_commands.register(subparsers) at mcp_server/cli.py:745 and driven by plugins/specflow2/skills/specflow-refine/SKILL.md.

Diagram

Single-screen visual — source in repo at docs/specflow-2.0/refinement-loop.html, which GitHub renders as source rather than as a page, so open the link or the local file. Testing walkthrough: docs/specflow-2.0/testing-the-refine-loop.md. Plan-vs-reality, including what was cut and what is still unproven: plans/specflow-2.0/status.md.

specs/ ──► /specflow-refine ⇄ /specflow-resolve ──► a spec with fewer holes
                                                    + the holes that remain

one round:

  refine new-round ──► grid: enumerate the decisions the spec implies, answer none
                            │
                            ▼
                       6 subagents, one lens each, no shared context
                            │  three passes each: name your own axes answering
                            │  nothing, fill every intersection, then re-read your
                            │  own matrix and account for every remaining cell
                            │  also fills the shared grid, writes reading.<lens>.json
                            ▼
                       coherence ──► can these answers all be true at once?
                            ▼
  refine round      ──► where readings disagree, with each lens's own phrasing
                        matrix cells a lens says the spec cannot answer, and why
                        matrix cells a lens enumerated and never came back to
                        grid cells nobody answered
                        grid cells every lens answered by guessing
                        blockers merged by id, with attribution
                        (no verdict, no score, no stop rule)
                            ▼
                       the skill judges: ask / assume / drop
                            ▼
  refine resolve    ──► decision recorded, spec file edited
                            │
                            └──► another round, or stop and say so as your own call

Details

Two plugins, and plugins/specflow/ is byte-identical to main. The refinement skills live in a new specflow2 plugin in the same marketplace. An earlier state of this branch de-symlinked specflow-analysis and specflow-planning and forked them, which meant two flows writing the same reserved filenames with content the 1.0 contract validator rejects; splitting the plugins fixed that at the root, and specflow plugin install installs specflow2 because it is the only one whose skills need this CLI present.

The interesting history is inside the branch. An early commit built roughly 2,850 lines of validators, a totality gate and a weighted ranking score, and a later commit deletes all of it. The gate was gameable and self-scoped, checking only the axes the agent under test had declared itself, so an artifact with every value marked guessed and sixteen undefined matrix cells passed clean. Reviewing the deleted code is optional, but the git log is the argument for the shape that remains.

There is deliberately no stop rule, which is the sharpest decision here. A round-to-round diff existed and reported what was new, and the skill read "nothing new" as convergence; that inference does not hold, because zero new blockers is equally consistent with this round's lenses simply finding less, and nothing holds lens effort constant between rounds. Turning it into a threshold would need the false-negative rate of a single round, which is unmeasured, so novelty, record_round, state.json and the new/repeat counts are gone and TestNoStopRule keeps them gone.

So completeness is reported as located incompleteness, never as a verdict. Disagreements say the spec did not determine this and name where, uncovered grid cells count gaps no reading even reached against an enumerated denominator, and agreed-but-guessed cells name consensus over silence — the failure a disagreement-only design structurally cannot see. Nothing sums them, because a found-something claim verifies one item at a time while a found-everything claim needs a denominator this design cannot have.

The per-lens matrix is what recovers building's one irreplaceable property, and plans/specflow-2.0/status.md §1a is the argument. A compiler never had to be told which decisions existed — it walked the graph and stopped at each one. Prose walks nothing and can omit a case without looking incomplete. So each lens names its own rows and columns before answering, then accounts for every intersection, and a cell it reaches and cannot answer is reported with its reason as a finding in its own right; that needs no second lens to corroborate, because "this intersection has no answer" is a fact about the spec rather than an opinion about it. Six lenses declare six cross-products from six angles, so a case invisible from one is often forced by another, and matrices are never merged across lenses because aligning one lens's axes with another's would take a similarity threshold.

Which is not the deleted gate, and the difference is the whole point. The gate scored an agent against axes the agent itself had declared, so declaring fewer axes passed it. The matrix declares axes too, but nothing passes, fails or blocks: unfilled cells are counted and printed, and a cell the lens skipped is reported as a hole in the reading rather than in the spec. The shared grid stays as well, narrower but comparable across lenses and the only thing here with a denominator. Both share the same bound: a decision no one's axes reach stays invisible, and no count reveals that.

Coherence contributes blockers and never votes. It runs after the lenses, reads all of their output, and asks which two answers cannot both be true, so compare() keeps it out of lenses and lens_count; counting it as a seventh independent reading would overstate the only evidence this design has.

One gap reads as one gap, but at most one. A disagreement at a location some lens already flagged attaches to that blocker rather than becoming a second entry, bounded at one per blocker so that N gaps under one coarse heading stay N items instead of collapsing into a single open blocker.

The filename decides a reading's lens. Concurrent subagents hand-write that field and a copy-pasted prompt yields two files claiming the same name, which would silently collapse them into one and take the disagreement between them with it; a mismatch is reported as a note rather than obeyed.

A malformed reading costs one lens, a malformed grid costs the round. Bad JSON or a wrong container shape in one reading is reported and the round continues with the rest, because six subagents write those concurrently; a broken grid refuses, because quietly treating it as absent would report a round with no coverage as a round with nothing to cover.

Models are the user's choice. 2.0 never calls a model, so the skills state the capability class each job needs and leave selection to the harness, and a harness that can route subagents to different providers can spread the lenses across model families.

Verification. 830 tests pass, 77 of them covering the refinement code, and the loop was driven end to end against the real CLI with hand-written artifacts. That walkthrough is written up as Level 1 of the testing doc so it is reproducible without a model, and running it shows the two channels agreeing independently: the concurrency lens reported seat hold × timer expiry as a cell the spec cannot answer, and the same underlying question surfaced separately as a cross-lens disagreement about who owns the timer.

What building still does that this does not. Two of its five properties are recovered at zero: contradiction by impossibility (a lens can argue two requirements conflict; nothing here fails to compile) and runtime discovery. The unbuilt specflow-contracts from plan §5 would recover the first cheaply by generating real SQL DDL or OpenAPI from the spec and running real validators; nothing short of building recovers the second, which is why step 7 tells the model to spike the one decision that is expensive, irreversible and split. status.md §1a scores all five.

Still unvalidated, and this is the thing to fix next. That disagreement between independent readings tracks real spec defects is the core hypothesis and has not been tested against a real spec, the agreed-guess and matrix signals inherit the same debt, and independence is unenforceable and currently unrecorded. Level 3 of the testing doc is the manual procedure, and until it has been run the honest claim is that the loop surfaces where independent readings diverged, not that it finds spec defects.

Scope. backend/, server.py and mcp_server/services/skills/ are untouched, the 1.0 flow is unchanged, and Steel Commandments I–XI still govern live backend code and come out with it rather than before. plans/specflow-2.0/specflow-plugin-plan.md now carries a banner marking sections 3 and 5 through 8 superseded by what actually shipped.