Skip to content

test(claude): pin the MCP complete_task validation gate coverage - #3343

Merged
huangruiteng merged 1 commit into
huangruiteng:mainfrom
NIU-123370:test/mcp-complete-task-validation-gate
Aug 21, 2026
Merged

test(claude): pin the MCP complete_task validation gate coverage#3343
huangruiteng merged 1 commit into
huangruiteng:mainfrom
NIU-123370:test/mcp-complete-task-validation-gate

Conversation

@NIU-123370

@NIU-123370 NIU-123370 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

GH-C85 / #3082 follow-up, now test-only and based directly on main.

What this pins

The MCP complete_task route reaches the caller-approved completion gate transitively: it invokes the real todo complete CLI, which runs run_completion_validation_gate before mutation. A future rewiring of the MCP path must not silently drop that machine-enforced obligation.

Coverage

The focused negative regression uses a real temporary registry/state and a real CLI subprocess (no monkeypatched subprocess.run). When a declared validation command fails, it proves that:

  • complete_task returns the typed validation_blocked_completion receipt;
  • the Todo remains open;
  • post-completion quota settlement is not attempted.

The earlier positive/no-command assertions were removed in response to review because they accepted failed settlement receipts as success. Positive settlement authority/order/identity belongs to blocked dependency #3342 and will be covered there after its runtime contract is corrected; this PR no longer contains or depends on that production change.

Verification

  • tests/test_goal_mode_mcp_completion_validation.py: 1 passed
  • tests/canary: 20 passed
  • focused Ruff check: passed
  • git diff --check: clean

Refs GH-C85, #3082.

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机

这个 PR 想补的缺口是成立的:MCP complete_task 不是自己实现 validation gate,而是经由真实 todo complete CLI 继承 run_completion_validation_gate;若以后重接 settlement 路径,确实可能把这条机器强制的完成门禁悄悄绕掉。负例使用真实临时 registry/state、真实 CLI 子进程,验证 declared command 失败时返回 typed validation_blocked_completion、Todo 保持 open 且不进入 settlement,这部分是有价值、可长期保留的回归覆盖。

改动思路

当前 exact head f93cd1575a9084fdf4262e4d397ef778973e5782 是叠加 PR:第二个提交新增 3 个端到端测试,第一个提交则完整带入 #3342_settle_completed_turn 运行时改动。正路径实际流程是 todo complete -> quota should-run -> quota spend-slot -> refresh-state;负路径在 todo complete 返回 typed block receipt 后停止。这里 authoritative state 应由同一 turn/Todo 的 durable writeback 与 spend receipt 共同证明,而不能只看到第一段 completion JSON 或只看到输出分隔符就判断链路成功。

具体改动

关键代码讲解

  • test_mcp_complete_task_fails_closed_on_failing_declared_validation:覆盖最重要的反例;检查 typed block receipt、Todo 未完成,以及 spend/refresh 均未触发。
  • test_mcp_complete_task_with_passing_declared_validation_commits:目前只经 _first_json_blob 读取第一段 completion JSON,并检查 Todo 状态;它没有读取或校验后续 settlement receipt。
  • test_mcp_complete_task_without_declared_command_keeps_fast_path:目前只断言 spend-slot/refresh-state 分隔符存在,没有断言两段 JSON 的 okappended、identity 或执行顺序语义。
  • _settle_completed_turn(来自堆叠的 #3342):先按 completion 后的新 selected_todo 执行 spend,再做无 typed delta 的 refresh;snapshot 解析失败时还降级为 unbound spend。这个运行时语义已经在 #3342 的 exact-head review 中被 REQUEST_CHANGES,本 PR 当前 diff 也包含它。

改动规模为 3 个文件、+357/-16;其中第二个提交是 158 行测试,第一提交是 51 行运行时代码加 148 行 mocked tests。活跃生产调用点是 MCP complete_task,所以这不是可按“test-only”忽略的 exact-head 风险。

对主干的风险

[P1 blocker] 绿色测试会接受 settlement 已失败的正路径,因而把当前错误 contract 钉成回归基线。 我在 exact head 上运行 uv run --extra test pytest -q tests/test_goal_mode_mcp_completion_validation.py,结果是 3 passed in 27.40s;随后用同一 fixture 打印 passing-validation 的完整输出:第一段 completion 为 ok: true,但 --- spend-slot --- 明确返回 ok: false(缺少 latest unspent accountable delivery writeback),紧接的 --- refresh-state --- 也返回 ok: false(open autonomous replan obligation 需要 typed semantic delta)。测试之所以仍绿,是 128–141 行只解析第一段 JSON,而 144–158 行只检查分隔符存在。

这意味着 positive/no-command walkthrough 没有验证它们声称保留的 post-completion settlement,远程 7 个成功 checks 也未覆盖这个语义。最小修复是:先让 #3342 修正 settlement authority/order/identity 并 rebase 本 PR;然后把每一段 receipt 解析出来,至少断言 refresh durable writeback 成功、spend 对同一 completed Todo/turn 的 accountable identity 成功,且 malformed/missing authoritative state typed fail closed,而不是 unbound fallback。若本 PR 只想覆盖 validation gate,可保留强负例,但不要用“分隔符存在”替代 settlement 成功。

Review lenses:validation block 使用 typed receipt,合格;文案保持 domain-neutral,合格;门禁被准确描述为 machine-enforced obligation,合格;但当前 exact head 的 runtime behavior 虽在正文标注为 stacked,测试结论仍掩盖了实际失败,因此 behavior-change validation 不合格。

我的整体评价

负向 validation-gate 用例的方向正确,也比 monkeypatched subprocess 覆盖更可信;问题不在测试框架,而在正路径 oracle 丢弃了大部分真实输出。当前 head 不能合入,因为它既包含已被 #3342 review 阻塞的运行时实现,又用会产生假绿的断言保护该实现。请在 #3342 修复后 rebase,并提交能证明完整 settlement receipts 成功的 exact-head 证据;届时我可以按新的 head 重新审。

English verdict: REQUEST_CHANGES — the negative validation-gate test is valuable, but the positive tests pass while both spend-slot and refresh-state receipts are ok: false, so this exact head pins a broken settlement path instead of validating it.

GH-C85 / huangruiteng#3082 follow-up: keep the review-approved real-control-plane negative regression proving that MCP complete_task inherits the declared-command validation gate. A failing command returns the typed validation_blocked_completion receipt, leaves the Todo open, and skips post-completion settlement. The branch is now test-only on main; positive settlement coverage remains with the blocked huangruiteng#3342 dependency instead of accepting failed receipts as success.

Signed-off-by: 牛瑞博 <912906590@qq.com>
@NIU-123370
NIU-123370 force-pushed the test/mcp-complete-task-validation-gate branch from f93cd15 to cbfdfa3 Compare August 21, 2026 02:58

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

动机

这个 PR 保护的是一条真实但容易被重构悄悄切断的机器门禁:MCP complete_task 本身不执行 Todo 的 caller-approved validation command,而是调用真实 todo complete CLI,由 complete_goal_todo() 在取得 mutation lock 前运行 run_completion_validation_gate()。如果以后 MCP 路径改成直接写 Todo、调整 CLI 入口,或者只根据 completion 输出继续 settlement,声明过的验证命令就可能被绕过。本 head 把范围收敛到这条最有价值的负向契约:验证失败必须返回 typed validation_blocked_completion、Todo 保持 open,并且不得触发后续 quota spend。相比旧 head,它移除了依赖 #3342 的运行时改动和会把失败 settlement 当成功的正向断言,因此原 blocker 已被实质修复。

改动思路

新 head cbfdfa34ddac5bd5654bec67c4c6e0cbc3035a74 直接基于 main,只增加一个真实控制面集成测试。测试建立临时 registry 与真实 ACTIVE_GOAL_STATE.md,通过 add_goal_todo() 写入带 failing validation command 的 advancement Todo,再让 GoalModeMCPControlPlane.complete_task() 调用真实 python -m loopx.cli todo complete。调用链为 MCP control → CLI → complete_goal_todo()run_completion_validation_gate();失败 receipt 使 _completion_succeeded() 返回 false,因此 complete_task() 不追加 quota spend-slot。最终同时从返回 payload 和 durable state 两侧验证 fail-closed 结果。

这条路径没有引入新的生产状态或抽象,authoritative state 仍是 Todo markdown 与 typed completion receipt。失败由 Todo completion gate 拥有,MCP 只透传结果;修复责任也保持在原 owner,而不是在 MCP adapter 里复制一套 validation 规则。

具体改动

本 PR 现在只有 tests/test_goal_mode_mcp_completion_validation.py,共 124 行测试代码,没有生产代码、公开协议或默认行为变化。

关键代码讲解

  • _write_fixture() 创建最小但真实的 active Goal、peer 注册信息和 runtime root,确保测试走 registry/state 解析而不是 mock 内部函数。
  • _add_todo() 使用产品 API 写入带 validation_command 的真实 advancement Todo,证明门禁来自持久化 Todo contract,而非测试临时拼装的 payload。
  • _control() 构造真实 GoalModeMCPControlPlane,只把命令前缀固定为当前解释器的 -m loopx.cli;没有 monkeypatch subprocess.run,因此 CLI 解析、进程返回和状态写回都在覆盖面内。
  • test_mcp_complete_task_fails_closed_on_failing_declared_validation() 验证四层结果:completion ok/completed 为 false、typed block receipt 存在、输出中没有 spend/refresh settlement 段、Todo durable status 仍非 done。
  • 生产调用点 GoalModeMCPControlPlane.complete_task() 只有在 _completion_succeeded() 确认 ok=truecompleted=truestatus=done 时才追加 spend;run_completion_validation_gate() 返回的 typed failure 不满足该条件,因而负向路径在 mutation 与 settlement 前关闭。

现有 tests/control_plane/test_todo_completion_validation.py 已覆盖 Todo gate 本身,但没有覆盖 MCP 经真实 CLI 的传递关系;现有 tests/test_kunluncode_goal_mode.py 覆盖 MCP completion 的一般行为,也没有声明命令失败场景。因此这不是重复 scaffold,而是对活跃 adapter 边界的耐久回归。

对主干的风险

没有发现阻塞问题。最强回归场景是未来修改 MCP completion 后,验证失败的 Todo 被错误标记 done 或继续消耗 quota;本测试同时检查 typed receipt、持久状态和 settlement 缺席,能直接捕获该问题。测试使用 subprocess,成本高于纯 unit test,但单用例约 5 秒,且这是证明跨 MCP/CLI 边界所需的合理代价。

typed-state lens 通过:失败由 validation_blocked_completionvalidation.passed=false 表达,没有 substring denylist 或 prose-only 分类。domain-neutrality 通过:fixture 与断言使用通用 Goal/Todo 语义。behavior-change disclosure 不适用,因为本 head 不改生产默认行为。guidance-vs-obligation 通过:测试和 docstring 都明确把 declared validation 描述为 machine-enforced gate,并验证忽略它时不得写终态或结算。

独立验证结果:该测试单独通过;与 MCP 及 Todo completion validation 的相关测试合计 57 passed;Ruff 通过;tests/canary 的 20 个唯一用例均通过。最初在无 .git 的 exact-head 归档里有两个 maintainability 用例因 git ls-files 环境前提失败,补上只读仓库索引后 maintainability 文件 9/9 通过,确认不是 PR 回归。GitHub 的 pytest、Windows、DCO 和 dependency review 也全部绿色。

我的整体评价

我批准 exact head cbfdfa34ddac5bd5654bec67c4c6e0cbc3035a74。作者准确响应了上一轮 review:删除叠加的 blocked runtime change、删除会接受失败 receipt 的正向 oracle,只保留已被认可的真实负向边界测试。当前改动单一目的、存在活跃生产调用点、验证覆盖正好落在 MCP 到 Todo gate 的传递风险上,且没有把 #3342 尚未解决的 settlement contract 偷渡回来。剩余风险仅是 subprocess 测试成本,规模可接受,不阻塞合并。

English verdict: APPROVE exact head cbfdfa34ddac5bd5654bec67c4c6e0cbc3035a74; the author removed the stacked runtime change and false-positive settlement assertions, leaving one durable real-CLI regression that proves a failing declared validation command keeps the Todo open and prevents quota settlement. Focused tests, canaries, Ruff, and GitHub CI are green.

@huangruiteng
huangruiteng merged commit b3a004d into huangruiteng:main Aug 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants