test: 9 control-plane tests fail on main (--required-read-id drift in evidence-log fixtures)

[NIU-123370]

Summary

9 control-plane tests fail on current main (verified at 943b4976, and also at 0e1aa649). They are unrelated to any open PR I'm aware of — surfacing separately so they don't get mistaken for regressions in other work.

Failing tests (9)

tests/control_plane/test_replan_evidence_tool_behavior.py (8):

• test_real_tool_loop_observes_production_evidence_log_intent
• test_tool_loop_accepts_a_bounded_evidence_read_plan
• test_tool_loop_accepts_supplemental_replan_observation_intent
• test_tool_loop_rejects_state_change_bundled_with_evidence
• test_tool_loop_rejects_unsafe_command_bundled_with_evidence
• test_tool_loop_accepts_the_real_host_utc_clock_shape
• test_tool_loop_allows_distinct_normal_workspace_preflight_reads
• test_tool_loop_rejects_generic_or_wrong_evidence_read

tests/control_plane/test_actual_default_model_behavior_portfolio.py (1):

• test_portfolio_real_tool_scenarios_choose_from_latest_quota_result

Repro

pytest tests/control_plane/test_replan_evidence_tool_behavior.py \
       tests/control_plane/test_actual_default_model_behavior_portfolio.py -q
# -> 9 failed, 18 passed

Root cause (diagnosis)

loopx/control_plane/runtime/agent_scoped_evidence_log.py:272 appends --required-read-id <id> to the evidence-log required-read command when a required_read_id is present:

if safe_required_read_id:
    parts.extend(["--required-read-id", safe_required_read_id])

The replan/portfolio scenarios build/expect the evidence-log required-read command without --required-read-id, so the parity/drift check raises and the turn resolves as quota_execution_failed, preempting the classification each test asserts (e.g. qualification_passed is True, or wrong_evidence_log / unexpected_command).

Notes

• Confirmed deterministic and not environmental: the same 9 fail identically on 0e1aa649 with an unrelated feature branch excluded.
• I came across this while preparing feat(todos): wire caller-approved validation_command into todo completion #3142 (for [Feature]: wire caller-approved validation into agent self-reported completion #3082); my change touches none of these files (git diff against the base is empty for control_plane/quota, control_plane/runtime/agent_scoped_evidence_log.py, and the two test files).
• I haven't fixed it — wanted to flag it cleanly first. Happy to take it on if the expected fix is "update the fixtures to include --required-read-id" (vs. a production change).

[NIU-123370]

Dug deeper — the --required-read-id drift splits into two layers. Layer 1 is a clean structural fix (below); Layer 2 is a golden-test refresh that's your design call.

Layer 1 — comparison sites (structural fix; fixes 9 → 2)

The drift check at control_plane/testing/replan_evidence_tool_behavior.py:725 did exact-string equality between the packet's required-read command and fixture.required_evidence_command. Since required_read_id is a dynamic content hash (replan-<sha16> of the replan obligation — autonomous_replan_obligation.py:81), pinning its bytes is the wrong check; structure + presence/shape is. Same for the direct assertion at tests/control_plane/test_replan_evidence_tool_behavior.py:117.

A small reusable helper makes both structural (I have this locally, applied at both sites — 9 failures drop to 2):

_REQUIRED_READ_ID_PATTERN = re.compile(r"^replan-[0-9a-f]{16}$")

def _strip_required_read_id(tokens: list[str]) -> list[str]:
    stripped, skip = [], False
    for t in tokens:
        if skip:
            skip = False; continue
        if t == "--required-read-id":
            skip = True; continue
        stripped.append(t)
    return stripped

def required_read_command_matches(fixture_command: str, packet_command: str) -> bool:
    p = _loopx_command_tokens(packet_command)
    f = _loopx_command_tokens(fixture_command)
    if p is None or f is None:
        return False
    rid = _argument_value(p, "--required-read-id")
    return (
        _strip_required_read_id(p) == _strip_required_read_id(f)
        and isinstance(rid, str)
        and bool(_REQUIRED_READ_ID_PATTERN.match(rid))
    )

I prefer asserting the id's shape (replan-<16 hex>) rather than its value — the value is supposed to move with the obligation's content, so an exact match is a false-positive trap.

Layer 2 — golden refresh (your call; the remaining 2)

These reflect new production behavior and the correct "after" state encodes your design intent for the replan-evidence flow, so I left them:

1. test_real_tool_loop_observes_production_evidence_log_intent at test_replan_evidence_tool_behavior.py:135 — rollout_events[-1]["event_kind"] == "quota_should_run" now sees evidence_log_read as the last event (production records an evidence_log_read rollout event after the quota step, a side effect of the required-read flow). → Should the golden last-event become evidence_log_read, or is that event unexpected here?
2. test_portfolio_real_tool_scenarios_choose_from_latest_quota_result — scenario turn_required_vision_replan classifies as wrong_evidence_log; its scripted evidence-log command no longer satisfies the classification now that the required command carries --required-read-id. → Needs the scenario script updated to your intended shape.

If you tell me the intended golden state for those two, I'll open a PR with Layer 1 + Layer 2 together. Otherwise the helper + structural drift check above are yours to fold into one comprehensive fix.

[huangruiteng]

在最新 main（8b8e7b248，2026-08-14）上核查：这个 9 个测试失败的问题已经不存在了，不是被修掉，而是被后续重构替换掉了。

• 8 个失败测试所在的 tests/control_plane/test_replan_evidence_tool_behavior.py 已在 feat(control-plane): close replan through semantic progress #3161（feat(control-plane): close replan through semantic progress）中删除，替换为 test_replan_semantic_action_behavior.py + loopx/control_plane/testing/replan_semantic_action_behavior.py；你提到的精确字符串 drift check 位置（control_plane/testing/replan_evidence_tool_behavior.py:725）也随之消失。
• 剩下的 test_portfolio_real_tool_scenarios_choose_from_latest_quota_result 仍在，且在当前 main 上通过（该测试也在 feat(control-plane): close replan through semantic progress #3161 / fix(control-plane): make fine execution use coherent decision slices #3148 中更新过）。
• --required-read-id 目前只出现在生产命令构造处（evidence_log CLI 和 agent_scoped_evidence_log.py），测试侧已没有对动态 hash 做逐字节黄金比对；required_read_id 的存在性/形状断言仍保留在 test_goal_vision_blocked_successor.py 等测试里。
• 实测：test_replan_semantic_action_behavior.py + test_actual_default_model_behavior_portfolio.py + test_agent_scoped_evidence_log.py 共 33 个测试全部通过。

结论：可以安全 close。你在评论里提出的 Layer 1 原则（比对结构/形状而不是动态值）与当前新套件的方向一致；Layer 2 的两个 golden 决策已被 semantic-action 重写取代，不再需要单独处理。

---

Checked on current main (8b8e7b248, 2026-08-14): the 9 failures no longer exist — not fixed in place, but replaced by a rewrite.

• The 8 tests lived in tests/control_plane/test_replan_evidence_tool_behavior.py, which was deleted in feat(control-plane): close replan through semantic progress #3161 (feat(control-plane): close replan through semantic progress) and replaced by test_replan_semantic_action_behavior.py + loopx/control_plane/testing/replan_semantic_action_behavior.py. The exact-string drift check you located (control_plane/testing/replan_evidence_tool_behavior.py:725) is gone with it.
• test_portfolio_real_tool_scenarios_choose_from_latest_quota_result still exists and passes on current main (it was also updated in feat(control-plane): close replan through semantic progress #3161 / fix(control-plane): make fine execution use coherent decision slices #3148).
• --required-read-id now appears only in production command construction (evidence_log CLI, agent_scoped_evidence_log.py). Tests no longer golden-match the dynamic hash byte-for-byte; presence/shape assertions for required_read_id remain in test_goal_vision_blocked_successor.py and friends.
• Ran locally: test_replan_semantic_action_behavior.py + test_actual_default_model_behavior_portfolio.py + test_agent_scoped_evidence_log.py → 33 passed.

Safe to close. Your Layer-1 principle (compare structure/shape, not the dynamic value) matches the direction of the new suite; the two Layer-2 golden decisions were superseded by the semantic-action rewrite, so no separate follow-up is needed.

[NIU-123370]

Closing this — thanks for the thorough verification. The failing suite was superseded by the semantic-action suite in #3161, and it's good to see the shape-over-value principle (compare modulo the dynamic replan id, assert its shape) live on there. Nothing left to fix here.