Skip to content

Agent-mode cost/provenance union path has zero test coverage (#545 landed at 63.4% coverage on new code) #615

Description

@sunholo-voight-kampff

Follow-up from mission-control iteration 158, which unblocked and merged #545 (squash 32583be57). Filing so it is not lost. Non-blocking — the evaluator scored the merge PASS 77/100 with zero blocking findings and all four required CI contexts passed.

Two independent instruments landed on the same gap, which is why it is worth a row.

1. The merged union path in agent_runner_multi.go is untested

RunAgentBenchmarkWithExecutor (internal/eval_harness/agent_runner_multi.go:463-479) recomputes cost from banked tokens when the executor self-reports $0, and banks CostProvenance unchanged from the executor:

costUSD := result.CostUSD
if costUSD == 0 {
    if c := CalculateCostWithBreakdown(lookupKey, result.InputTokens, result.OutputTokens+result.ReasonTokens); c > 0 {
        costUSD = c
    }
}
// ...
Cost:           costUSD,
CostProvenance: string(result.CostProvenance),

This is correct, and the reason is worth writing down because it is not obvious from the code: provenance is a property of the auth lane, not of who did the arithmetic, so recomputing the number must not change the label. ResolveCostProvenance (internal/executor/cost.go:139) is total — every branch returns a named value with an explicit CostProvenanceUnknown default — so an empty/unlabelled provenance is impossible by construction, and all six executors call it (verified: claude, codex, managed_agents, motoko, opencode, pi).

The problem is that nothing tests it. Positive control that the search instrument works — it does find a reference:

$ grep -rn 'RunAgentBenchmarkWithExecutor' --include='*_test.go' .
internal/eval_harness/agent_verify_test.go:213:// ... RunAgentBenchmarkWithExecutor (the live multi-executor path,

That single hit is a comment, not a call. No test invokes the function or exercises lines 463-479. cost_provenance_test.go covers standard-mode provenance and JSON round-tripping only. So the reasoning above is load-bearing and unguarded: a future refactor that "helpfully" recomputes provenance alongside the cost would red nothing.

Asked for: a unit test pinning the executor-reports-zero case — cost recomputed, provenance preserved — so that the invariant is a test rather than a paragraph.

2. Sonar: 63.4% coverage on new code (required ≥ 80%)

SonarCloud Code Analysis failed on #545's head with 63.4% coverage on new code. Note this is a different number from dev's standing 78.7%, so it is PR-attributable rather than inherited — checked rather than assumed, since dev has carried its own Sonar red for six consecutive analysed commits. Sonar is non-required (UNSTABLE is not BLOCKED), which is why the merge proceeded, but 63.4% on ~1,500 added lines is the same finding as item 1 seen through a second instrument.

3. pi cannot classify a subscription lane

internal/executor/pi/pi.go:379,416 both hardcode executor.ResolveCostProvenance(task, executor.AuthLaneBilled); there is no authLane() method in the package at all. Compare claude (reads AILANG_AUTH_MODE) and codex (reads ~/.codex/auth.json).

Not an active defect: pi is currently used only through API-key OpenRouter lanes, where metered is truthful. It becomes one the moment a pi model is pointed at a subscription-authenticated lane, at which point real spend and list-price-equivalent get summed under one label — exactly the defect #545 exists to fix, reintroduced through the one executor added after that branch was cut.


Found by mission-control iteration 158 · charter ## STATUS … ITERATION 158

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions